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Executive Summary 

A. Purpose 

The 1992 Teacher Follow-up Survey (TFS) Reinterview and Extensive 
Reconciliation attempted to uncover why respondents’ answers differed 
between the original TFS and the reinterview. It differed from any other 
Census Bureau reinterview (including previous TFS reinterviews designed to 
exclusively measure response variance estimates). The essential difference 
was that it employed an extensive structured reconciliation. This report, the 
first of a two-part report, presents the results of the reinterview and extensive 
reconciliation, and offers suggestions for improving the questionnaires. The 
second part discusses the reinterview and reconciliation methodology and 
makes recommendations for improving it. 

B. Major Findings 

1. A subset of questions from both the Current Teacher (TFS-3) and the 
Former Teacher Follow-up Survey (TFS-2) were chosen for reinterview 
and reconciliation. A comparison of the before-reconciliation L-fold 
indexes of these questions (i.e., measures derived from the reinterview 
before the questions were reconciled) indicates that respondents had 
difficulty providing consistent answers to the following questions: 

TFS-3: 3b Teaching Assignment 
TFS-3: 7a Teaching Certificate in Other Field 
TFS-3: 31 Total Combined Income 
TFS-2: 32a Other Dependent Persons. 

These four questions had before-reconciliation L-fold indexes in the 
moderate range. The other questions either had indexes in the low 
range or did not meet the minimum requirements necessary to compute 
a reliable estimate. 

2. In general, the 1992 TFS Reinterview and Extensive Reconciliation 
failed to provide adequate reasons for differences between the original 
and reinterview responses. We obtained a limited number of reasons 
because the reinterview failed to detect the same level of differences in 
respondents’ answers as previous reinterviews. This was a 
disappointing result and, in large part, attributable to the fact that the 
1992 TFS Reinterview and Extensive Reconciliation employed a 
dependent-type reinterview. We discuss this result in detail and offer 
suggestions for improving the methodology in the second part of this 
report. 
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C. Recommendations 

1. We suggest rearranging and rewording the answer categories for the 
teaching assignment question (TFS-3, 3b) as follows: 

You provide instruction at more than one school (e.g., 

you are an itinerant, traveling, co-op, or satellite 
teacher). 

You fill the role of a regular teacher on a long-term 

basis, but you are still considered a substitute (i.e., 
you are a long-term substitute teacher). 

You are a regular full-time or part-time teacher. 

This rearrangement gives respondents the choices of "itinerant" and 
"long-term substitute" before the choice of "regular," which should 
reduce the number of inappropriate choices of "regular." This woiding 
also prominently displays the definitions for itinerant and substitute and 
minimizes technical terminology. 

2. We also suggest rearranging the TFS-3 questions as follows: 

5a. Main Teaching Assignment Field 

6a. Teaching Certificate in Main Field 

6b. Type of Certificate 

6c. Certificate Granted within last 12 months 

5b. Teach Classes in Other Assignment Fields 

7a. Teaching Certificate in Other Field 

7b. Type of Certificate 

7c. Certificate Granted within Last 12 Months. 

With this arrangement, a "no" response to question 5b will cause a skip 
to question 8. This skip pattern improves question 7a by eliminating 
the troublesome "not applicable" answer category. 

3. We need further research before we can suggest improvements to the 
income questions. (This includes question TFS-3, 31.) The results of 
this and other research suggest that we need to gain a better 
understanding of respondents’ use of records. Also, we need to 
determine just how much measurement error in the income data the 
sponsor can tolerate. 
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4. We have no recommendation for question TFS-2, 32a. However, we 
provide suggestions for a number of other questions in Detailed 
Recommendations, Section IV. 

11- Methodology 

A. Description of the TFS Reinterview 

The Census Bureau conducts the Schools and Staffing Survey (SASS) for the 
National Center for Education Statistics. The SASS is a relatively new set of 
integrated surveys first launched in the 1987-88 and 1990-91 school years, and 
scheduled every three years hence. Currently, ten self-administered 
questionnaires designed to obtain nationwide information on teacher and 
student counts, administrator and teacher backgrounds, as well as other 
administrator, school, and teacher-level characteristics comprise the core of the 
SASS. 

A year after collecting information from the teachers in the SASS, the Census 
Bureau conducts two more surveys with sub-samples of the teachers. The first 
is the TFS, closely followed by the second, the TFS Reinterview. Both 
surveys contain a former teacher and a current teacher component. The 
former teacher component asks questions of respondents who have left the 
teaching profession since responding to the SASS. (Before selecting the 
sample for the TFS, the Census Bureau contacts the schools to determine 
which teachers have left teaching since responding to the SASS.) The current 
teacher component asks questions of those who have remained in the teaching 
profession. 

The TFS’ purpose is to provide information about teacher attrition and to 
project teacher demand (Faupel et al., 1992). The TFS, like any survey, is 
subject to measurement error. The purpose of the 1989 TFS Reinterview was 
to expressly measure one of these errors: response variance of selected 
questions (Waite, 1990). While measuring response variance exposed 
inconsistencies in respondents’ answers between interviews, it did little to 
explain why the inconsistencies occurred. Therefore, the 1992 TFS 
Reinterview and Extensive Reconciliation was designed to ascertain why 
respondents’ answers differed. The ultimate goals were: 1) to determine if . 
respondents’ answers differed because they were having difficulty 
comprehending the questions or the response tasks asked of them, and 2) to 
make recommendations for correcting these difficulties. 

B. Sample Selection 

The 1992 TFS Reinterview and Extensive Reconciliation included 
approximately 700 current teachers and 800 former teachers drawn from the 
TFS sample file. Working from a paper questionnaire, supervisory field 
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representatives (SFRs) administered the TFS Reinterview and Extensive 
Reconciliation by phone in the spring of 1992. 
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The completion rate is the number of completed reinterviews divided by the 
number of eligible reinterview cases. The 1992 TFS Reinterview and 
Extensive Reconciliation achieved a 92 percent completion rate compared to an 
81 percent rate in 1989. The 1992 rate is significantly greater than the 1989 
rate at the 90 percent confidence level. Table 1 shows the final status of the 
1992 reinterview sample. 



Table 1. Reinterview Sample and Status 



Status 


Total 


Former 

Teachers 


Current 

Teachers 


Selected for Reinterview 


1498 


790 


708 


Eligible for Reinterview 


1425 


747 


678 


Completed Reinterview 


1314 


685 


629 


Switched Leaver/Stayer 


17 


11 


6 


Leavers 


666 


664 


2 


Stayers 


631 


10 


621 


Unable to Contact 


59 


34 


25 


Refused 


34 


15 


19 


Other Noninterview ' 


18 


13 


5 


Ineligible for Reinterview 
Original Noninterview 


73 


43 


30 



C. Reinterview Procedure 

The 1992 TFS Reinterview and Extensive Reconciliation questionnaires were, 
designed to identify and reconcile differences between the original survey and 
the reinterview. All reinterview questions were asked before any of the 
reconciliation questions. It was decided that this approach would affect the 
reinterview responses less than if the reconciliation questions were asked 
throughout the reinterview. 



1 Four regional offices neglected to reinterview 18 cases (ten in Philadelphia, four in 
Charlotte, two in Atlanta and two in Los Angeles). We realized this late in June after 
closeout. 
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Reinterviewers were instructed in a home self-study manual to first read 
through all of the reinterview questions at the top of the left-hand pages. 

When they were done with this component of the interview, they were told to 
go back to the beginning of the questionnaire, and to compare respondents’ 
reinterview responses with their original responses. Original responses were 
transcribed at the top of the right-hand pages. (These responses were not 
covered over, but clearly visible during the reinterview.) When a difference 
between the two responses occurred, reinterviewers were instructed to ask 
reconciliation probes designed to elucidate the reasons for differences in 
respondents’ answers. These probes were printed on the questionnaire beneath 
the original response on the right-hand side of the page. 

D. Departures from an Ideal Model for Response Variance 

The ideal model of a response variance reinterview assumes a model of 
independent replication. With this model, responses have the same expected 
values across trials. The expectation is that respondents asked the same 
questions under the same conditions will repeatedly provide the same answer, 
unless a real change has occurred in the characteristic being measured. Given 
independent replication, differences in respondents’ answers suggest that at 
least one failure occurred at some point in the question-response process 
during at least one of the interviews. Either the questions: (1) were 
misunderstood; (2) required information the respondents did not have, could 
not recall, or could not provide in the detail desired; or (3) asked for 
information respondents were not willing to accurately provide. Given 
independent replication, questions with high levels of differences exhibit these 
problems to a greater degree than questions with low levels. - As a result, 
questions exhibiting high levels of differences are of the greatest concern to us 
because they contribute the most to measurement error. 

The design of the 1992 TFS Reinterview and Extensive Reconciliation 
departed from the ideal in four ways. First, the original TFS responses were 
transcribed onto the reinterview questionnaires. Previous research 
demonstrates that providing the original responses on the reinterview 
questionnaire has a biasing effect on the reinterview responses. Furthermore, 
this research indicates that the index of inconsistency, a measure of response 
variance, is depressed even further when reconciliation questions are asked in 
addition to the reinterview questions (Koons, 1973). 

A second way in which the TFS Reinterview and Extensive Reconciliation 
differed from the ideal is that the original interviewers were field 
representatives, while the reinterviewers were supervisory field 
representatives. In this case, response variance may reflect interviewer 
differences, rather than purely reflecting respondent or instrument error. 
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A third way in which the design departed from the ideal is that the 1992 TFS 
Reinterview and Extensive Reconciliation was administered exclusively by 
phone, whereas 56 percent of the original cases were self-administered. 

Concern about response effects due to mixed modes of administration has 
taken on new meaning as surveys increasingly move towards mixed modes of 
collection. At this time, the literature (e.g., Tamai and Dillman, 1989; 

Dillman and Tamai, 1990; Parmer et al., 1992) seems to suggest that 
questions concerning "subjective" phenomena are more likely to exhibit mode 
response effects than questions concerning "objective" phenomena. Since the 
reinterview questions for the 1992 TFS are mainly "objective" in nature, the 
mixed mode of administration should have less of an effect on the results than 
might otherwise be the case. 

A fourth way in which the 1992 TFS Reinterview and Extensive Reconciliation 
departed from the ideal, and this would be true for any reinterview, lies with 
the effect respondents’ memories play in the process. Respondents may wish 
to appear consistent, rather than admit they misunderstand something. To the 
extent this occurs, reinterview measures will be artificially depressed, leading 
one to erroneously conclude the questions are relatively free from problems. 
Blair and Sudman (1993) found between one-third to one-half of reinterview 
respondents reported trying to remember their original answers. Whether or 
not this led to artificially suppressed reinterview measures was not studied. 

E. Questionnaire Development 

Two methods were used to develop the extensive dose-ended reconciliation 
questions. First, an expert analysis was conducted in which potential problems 
with the selected survey questions or possible reasons for differences between 
the two interviews were identified (see Forsyth and Lessler, 1991, for a 
discussion of this method). This method often yielded probes that were 
directed at determining if the differences between the two interviews reflected 
real change. Of course, this kind of finding would not indicate a problem with 
the questionnaire’s design. An example of this situation would be if a 
respondent said he or she was a teacher in the original interview, but said he 
or she was £& a teacher in the reinterview. In such a case, respondents were 
asked if they had left teaching since responding to the TFS, and if so, when - 
they had left. The "when” question was used to determine whether the 
original or reinterview response was correct. 

The expert analysis was also used to identify problems respondents might have 
understanding questions, recalling information, disclosing information, and so 
forth. If, for example, respondents reported the reverse of what they reported 
above, they were asked if they were temporarily not teaching at the time of the 
original interview (if, for instance, they were on spring break, vacation, sick 
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leave, or temporarily not teaching for some other reason). This would indicate 
that they misunderstood the TFS’ definition of teaching. 

Secondly, the findings of previous cognitive research with the 1990 Field Test 
Teacher Questionnaire (see Bates and DeMaio, 1990) were used to develop 
closed-ended probes. This information was especially helpful in identifying 
questions that might be susceptible to misinterpretation. For instance, the 
1990 Teacher Questionnaire, the 1992 TFS, and the 1992 TFS Reinterview 
and Extensive Reconciliation all asked: "What is your MAIN teaching 
assignment at this school, that is, the field in which you teach the most 
classes?". Respondents are supposed to choose one field from a list of 54. 
Cognitive research revealed that special education teachers had a hard time 
choosing between the special education students (e.g., Deaf and Hard-of- 
Hearing) and the subjects they taught to these students (e.g., Physical 
Education/Health Education). Probes were developed in instances like this to 
expose respondents’ ambiguities. 

When a discrepancy occurred between a respondent’s original and reinterview 
response, and the respondent had not chosen one of the closed-ended 
responses, the reconciliation of the item ended by asking an open-ended 
question: "Or was there some other reason [for the difference]?". The open- 
ended responses were compiled and subsequently coded. Statisticians 
developed the coding scheme. After coding approximately one-third of the 
write-in responses as examples and developing coding specifications, clerks in 
the Data Preparation Division coded the remainder. 

F. Analysis 

We computed three measures to analyze the data from the TFS Reinterview 
and Extensive Reconciliation: gross difference rate (GDR), net difference rate 
(NDR) and index of inconsistency. The GDR and the index are measures of 
response variance. The NDR is a measure of response bias. However, the 
1992 TFS Reinterview and Extensive Reconciliation deviated from the 
assumptions necessary to ideally measure both response variance and bias. 
Therefore, our estimates of response variance (GDR and index) are greatly 
understated, while the effect on our estimate of response bias (NDR) is 
unknown. 

1. Gross Difference Rate (GDR) 

The GDR is the percentage of respondents who change their answers 
between the original and the reinterview. It is difficult to interpret as 
an exact measure. Rather, it provides only a rough idea of how 
consistently respondents answer a question. A large GDR shows 
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serious response variance in the data. It signals that the question does 
not provide reliable information. However, a small GDR does not 
guarantee high consistency in responses. The magnitude of its value 
depends on the question’s content and expectation. 

We compute a GDR before reconciliation and after reconciliation for 
each question. For a question with two possible answer categories, 
consider the 2x2 table 



Reinterview 

Responses 



Answer 
Category 1 

Answer 
Category 2 

The GDR » (b+c)/n. 



Original Responses 

Answer Answer 

Category 1 Category 2 

a b 



c d n— a+b+c+d. 



2. Net Difference Rate (NDR) 



The NDR measures response bias using reconciled data. It tells us if 
we have an overcount or undercount in the original interview for a 
particular answer category. If the NDR for an answer category is 
significantly different from zero, this signals a problem with the answer 
category, that is, a bias exists. If the NDR is greater than zero, the 
category was over-reported in the original interview. If the NDR is 
less than zero, the category was under-reported in the original 
interview. 



If none of the NDRs for a question’s answer categories are significantly 
different from zero, then there is no evidence that a particular answer 
category creates a bias. 

We compute a NDR after reconciliation for each answer category for a 
question. For the above 2x2 table, the NDRs are 




( 



Answer Category 1 
(c-b)/n 



Answer Category 2 
(b-c)/n. 
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3. Index of Inconsistency and L-fold Index 

In a strictly independent reinterview, the index of inconsistency 
measures response variance for an answer category of a question. We 
compute the index using unreconciled data. For the above 2x2 table, 
the index for category 1 is 

I = bfC 

l/n[{(a+c)(c+d)} + {(a+b)(b+d)}] . 

The L-fold index measures the response variance for a question. It is 
the average of the indices for all the individual answer categories. We 
interpret both the index and L-fold as follows: 





Response 

Variance 


Question’s 


Index 


Level 


Interpretation 


less than 20 


low 


not a problem 


20 to 50 


moderate 


some problems 


over 50 


high 


major problems. 



Because the 1992 TFS Reinterview and Extensive Reconciliation was 
not independent, we use the L-fold index as a measure of relative 
rather than absolute response variance. In other words, we use the 
index solely as a gauge to direct us towards the more troublesome 
questions. Appendices A and B offer an inclusive list of these L-fold 
indexes for the reinterview questions by current teacher and former 
teacher component, respectively. 

III. Results 

A. Question-by-Question Review of the Current Teacher Follow-up Survey 
(TFS-3) 

In this section we present the data for the questions chosen for reinterview 
from the 1992 Current Teacher Follow-up Survey (TFS-3). We attempt an 
explanation of the data if a question appears troublesome because of a 
moderate L-fold index, one or more significant NDRs, or one or more notable 
reasons for differences. 

The number of responses listed for a question represents the total number of 
respondents who answered the question in both the original and the 
reinterview. All question numbers refer to the original (TFS-3), not the 
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reinterview (TFS-3R) questionnaire. We list the data for each question in a 
table containing up to three parts. 

• Part 1 lists the question’s GDRs before and after reconciliation. For 
monetary, state, month and year responses and check box information, 
a disagreement rate replaces a GDR. The disagreement rate does not 
consider answer categories but only whether the responses agree or 
diagree according to the reinterview instructions. Confidence limits at 
the 90% confidence level are below each GDR and disagreement rate. 

• Part 2 lists each after reconciliation NDR for an answer category that is 
significantly different from zero at the 90 percent confidence level. 
Corresponding confidence limits are below each NDR. We eliminate 
this part if a question has no answer category with a significant NDR. 
Appendix C offers an inclusive list of the after reconciliation NDRs for 
all answer categories by question. NDRs exist only for questions 
where we calculated a GDR. Questions with a disagreement rate have 
no answer categories and therefore no NDRs. . 

• Part 3 lists the reasons for differences between the question’s original 
and reinterview responses if one or more reasons stand out. We 
eliminate this part if the reason counts for the question are low and 
evenly distributed. Appendix D offers an inclusive list of all the 
reasons for differences by question. 

Please note that the total number of reasons for differences does not 
always equal the number of response differences for a question. Our 
data file contains cases where the reinterviewer entered a reason for a 
difference when the responses agreed or failed to enter a reason for a 
true difference. Discrepancies therefore exist between reason counts 
and difference counts. (We talk about this further in the second part of 
this report.) 

i 

One of the reasons for differences in many of the following tables is 
"Manual/General Error." It refers to write-in entries in which the 
respondent admitted to making some kind of mistake (e.g., "[I) first • 
marked 8, changed to 9, forgot to erase 8." or "I forgot one zero, I 
guess."). Based on respondents’ descriptions, these do not appear to be 
systematic errors; rather they seem to be random errors over which we 
could never hope to have control. 

Also, both "Misread Question" and "Misunderstood Question" are 
given as distinct reasons for differences in the tables. These are 
reasons respondents gave in answer to the open-ended question. We 
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left them as distinct, since respondents may have had different 
meanings in mind. To misread something suggests that a respondent 
failed to read the sentence as given (i.e., another word(s) registered in 
place of the one(s) given). This implies that respondents were capable 
of interpreting the sentence correctly if only they had read it correctly. 
This error may be impossible to correct if the respondent was simply 
being careless or was distracted. In contrast, the misunderstood 
category suggests that respondents were incapable of understanding the 
meaning of the given words, despite having read them correctly. A 
solution for this error, although not necessarily easy to identify, should 
exist. Please note, however, that these interpretations may or may not 
be true, since respondents were not probed further about either of these 
responses. 



Cover Question. Are you currently teaching - full-time, part-time, or as a 
long-term substitute - in grades K through 12? 



Table 2. Cover Question - 613 Responses 



No. of 
Categories 


Before Reconciliation 
GDR% 


After Reconciliation 
GDR% 


2 


1.0 

(0.3, 1.6) 


no responses change with 
reconciliation 



Question 1. Of the following, which is your MAIN activity and which, if 
any, is your OTHER main activity? MAIN and OTHER main activity is 
defined in terms of the amount of time you spend on each during the 
normal work week. If your time is usually divided between two 
occupations or activities, e.g., you are retired but have returned part-time 
to teaching, list either one as MAIN and the remaining one as OTHER. 

This question has three parts: main activity, "other* main activity, and a 
check box if time is equally divided between the two activities. Reconciliation 
involved only the main activity response. 

The NDR (-0.8%) in part 2 of Table 3 reveals that respondents tended to 
underreport "teaching in an elementary or secondary school" as their main 
activity in the original interview. Part 3 shows that nearly one-third (13) of 
the respondents who gave a reason for inconsistent answers said they didn’t 
know why they did so. Another third (12) said they misunderstood the 
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question. Respondents probably misunderstood this item because it is long and 
awkward. 

"Main and Other Main Activity” is an unusual way of asking someone what 
they primarily do and what, if anything, else they do. In fact, "Other Main 
Activity” is an oxymoron. Generally one thinks of main as the most 
important, so how can something be both classified as OTHER, meaning less 
than most important, and MAIN, meaning most important? 

Using the words ”Main” and ”Other Main Activity” probably came about as a 
means of avoiding the words "primary” and "secondary,” since cognitive 
research with elementary teachers revealed that they often associated the word 
"primary" with elementary grade levels (Bates and DeMaio, 1999). Also, the 
words "primary” and "secondary" suggest that a teacher’s time is unequally 
divided between the two activities, when in fact it may be equally divided. 
"Other" and "Other Main Activity" probably came about as a means of 
avoiding this suggestion. 
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Table 3. Question 1 (Main Activity) - 595 Responses 



Part 1. Gross Difference Rates and Confidence Limits (%) 


No. of 
Categories 


Before Reconciliation 


After Reconciliation 


7 


1.5 

(0.7, 2.3) 


1.2 

(0.4, 1.9) 


Part 2. Signifies 


int NDRs - After Reconciliation 


Answer Category 


NDR% 


Teach in elementary/secondary school 


-0.8 

(-1.6, -0.1) 


Part 3. Reasons for Difference between Responses 





Reason 


Count | 


Percent 


Total 


41 


100.0 


Don’t know 


13 


31.7 


Misunderstood question 


12 


29.3 


Time divided between answer categories 


4 


9.8 


Forgot/remembered info 


3 


7.3 


Situation changed since responding 


2 


4.9 


Manual/general error 


2 


4.9 


Category problems 


2 


4.9 


Other 


1 


2.4 


Misunderstood reference period 


1 


2.4 


Misread question 


1 


2.4 





Table 4. Question 1 (Other Main Activity) - 209 Responses 
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No. of 
Categories 


Before Reconciliation 
GDR% 


After Reconciliation 
GDR% 


7 


6.7 

(3.9, 9.5) 


no reconciliation 



Table 5. Question 1 (Time Equally Divided) - 629 Responses 



No. of 


Before Reconciliation 


After Reconciliation 


Categories 


Disagreement Rate (%) 


Disagreement Rate(%) 


check box 


4.8 


no reconciliation 


(2) 


(3.4, 6.2) 





Question 2. Which of the following categories describes your position as a 
school EMPLOYEE? 



Table 6. Question 2 - 619 Responses 



No. of 


Before Reconciliation 


After Reconciliation 


Categories 


GDR% 


GDR% 


5 


1.1 


no responses change with 




(0.4, 1.8) 


reconciliation 



Question 3a. How much time do you work as a TEACHER at this school? 



Table 7. Question 3a - 612 Responses 



No. of 
Categories 


Before Reconciliation 
GDR% 


After Reconciliation 
GDR% 


5 


1.5 

(0.7, 2.3) 


no responses change with 
reconciliation 
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Question 3b. Which of the following categories best describes your 
teaching assignment? 

Part 3 of Table 8 shows that half (6) of the respondents who gave a reason for 
inconsistent answers said they misunderstood the question or thought the 
answer categories were confusing. The NDRs in part 2 of Table 8 suggest 
that the problem lies with the first two answer categories: (1) Regular full- 
time or part-time teacher and (2) Itinerant teacher. Respondents tended to 
overstate being a regular full- or part-time teacher (1.6%) in the original 
interview, while understating being an itinerant teacher (-1.5%). 

There appear to be two possible explanations for this. In the first, respondents 
in the original interview read through all of the answer categories, but not the 
parenthetical information. In this case, they do not know the definition of an 
"itinerant" teacher. Although the definition is provided, they do not bother to 
read it. Instead, they ignore this category and choose one that does make 
sense to them and in which they can reasonably fit: the full-time or part-time 
teacher category. In this case, respondents are either overlooking, ignoring, 
or not fully understanding the use of the word "regular" in the first answer 
category. Since reinterviewers are trained to read the question and answer 
categories exactly as worded, it is likely that respondents pick up on their 
mistake and change their answers in the reinterview. 

In the second scenario, respondents choose the first answer category because 
they think it fits their situation well enough. Again, they probably overlook, 
ignore, or do not fully understand the word "regular." This case differs from 
the first in that here respondents do not bother to read the remaining answer 
categories. 

Finally, a piece of evidence supporting the notion that the word "itinerant" is a 
problem comes from previous research. Cognitive research with the Public 
School Questionnaire revealed that many respondents did not know what an 
"itinerant" teacher was (Jenkins et al., 1992, p. 26). They knew "itinerant" 
teachers by other names, including traveling, co-op, and satellite teachers. 
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Table 8. Question 3b - 610 Responses 



Part 1. Gross Difference Rates and Confidence Limits (%) 


No. of 








Categories 


Before Reconciliation 


After Reconciliation 


3 


2.0 


no responses change with 




(1.0, 2.9) 


reconciliation 


Part 2. Significant NDRs - After Reconciliation 


Answer Category 


NDR% 


Regular full/part-time teacher 


1.6 








(0.7, 2.6) 


Itinerant teacher 




-1.5 








(-2.4, -0.6) 


Part 3. Reasons for Difference between Responses 


Reason 


Count 


Percent 


Total 




13 


100.0 


Misunderstood question 


3 


23.1 


Category problems 


3 


23.1 


Situation changed since responding 


2 


15.4 


Don’t know 




2 


15.4 


Forgot/remembered info 


1 


7.7 


Manual/general error 


1 


7.7 


FR error 




1 


7.7 



Question 4. If you are a full-time school or district employee with less 
than a full-time teaching assignment, which of these best describes your 
other assignment at this school? 

Part 3 of Table 9a shows that nearly half (9) of the respondents who gave a 
reason for inconsistent answers said they didn’t know why. We thought that 
perhaps respondents had difficulty understanding the phrase "If you are a full- 
time school or district employee with less than a full-time teaching 
assignment.” As a result, we hypothesized that respondents neglected to 
correctly mark the *not applicable” answer category in one of the interviews 
and marked the "other” box and wrote in *full-time teacher” in the other 
interview. 
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We looked at the data more closely to see if this was the case. Table 9b 
shows that, indeed, respondents had trouble with the "not applicable" answer 
category. All fourteen respondents who answered this question inconsistently 
marked the "not applicable" category in one of the interviews. However, the 
data reveal that only four respondents marked the "other" box in the other 
interview. The vast majority of respondents (10) specified a particular 
assignment. 

Assuming that respondents wouldn’t mistakenly report being an administrator 
(for instance) in one of the interviews if in fact they weren’t actually an 
administrator, these data suggest that respondents are incorrectly marking the 
"not applicable" category in one of the interviews. This mistake still implies 
that respondents are having difficulty understanding the question, and we still 
think that this difficulty is related to misinterpreting the phrase "If you are a 
full-time school or district employee with less than a full-time teaching 
assignment." 



Table 9a. Question 4 - 518 Responses 



Part 1. Gross Difference Rates and Confidence Limits (%) 


No. of 








Categories 


Before Reconciliation 


After Reconciliation 


8 


2.7 


no responses change with 




(1.5, 3.9) 


reconciliation 




Part 3. Reasons for Difference between Responses 


Reason 


Count 


Percent 


Total 




20 


100.0 


Don’t know 




9 


45.0 


Missed skip pattem/question 


3 


15.0 


More than one category fits situation 


2 


10.0 


Wasn’t sure what one/more categories meant 


2 


10.0 


Misunderstood question 


2 


10.0 


Other 




1 


5.0 


Category problems 


1 


5.0 
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Table 9b. Distribution of Interview and Reinterview Response Differences 
to Question 4 



Interview Response 


Reinterview Response 


Count 


Total 


Total 


14 


An Assignment 


Not Applicable 


6 


Not Applicable 


An Assignment 


4 


Other 


Not Applicable 


2 


Not Applicable 


Other 


2 



Question 5a. What is your MAIN teaching assignment at this school, that 
is, the field in which you teach the most classes? 

This question has two parts: main assignment field and a check box if the 
respondent’s teaching assignment is equally divided between fields. 
Reconciliation involved only the main assignment field. To answer this 
question, the respondent chose a teaching assignment field (e.g., Biology) from 
a list of 54 fields. The list was divided into seven subject areas (e.g., 

Science). To ease analysis, we collapsed the 54 fields into the seven subject 
areas. Attachment 1 lists the 54 fields by subject area. 

The GDR (1.6 %) in Table 10 shows that for the most part respondents 
consistently chose a field from within the same subject area (e.g., they chose a 
science field in both interviews). Please note, however, that this is not the 
same as saying they chose the same field both times. Because we collapsed 
the fields, we are not capable of discerning whether they chose the same field. 
At this level of analysis, this question appears to be relatively trouble free. 



Table 10. Question 5a (Main Teaching Assignment) - 617 Responses 



j No. of 

1 Categories 


Before Reconciliation 
GDR% 


After Reconciliation 
GDR% 


7 


1.6 

(0.8, 2.5) 


1.5 

(0.7, 2.3) 
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Table 11. Question 5a (Teaching Assignment Equally Divided) - 629 Responses 



No. of 
Categories 


Before Reconciliation 
Disagreement Rate (%) 


After Reconciliation 
Disagreement Rate(%) 


check box 
(2) 


3.0 

(1.9, 4.1) 


no reconciliation 



Question 5b. Do you teach classes in OTHER fields at this school? 

If respondents answered "yes" to this question, they were asked a follow-up 
question: In what field do you teach the second most classes? To answer 
this question, the respondent chose from the same 54 fields listed for Question 
5a. We again grouped these fields into the same seven subject areas for 
analysis. 

The NDR (-3.4%) in part 2 of Table 13 reveals that respondents tended to 
underreport the "special areas" categories in the first interview. The reason 
for this is not clear. The GDR (3.4%) for this question is greater than that for 
the previous comparable question (1.6%). However, part 3 of Table 13 shows 
that 40 percent (4) of the respondents reported legitimate reasons for 
answering this question inconsistently. They either taught both fields and 
reported one in the original and the other in the reinterview (3), or they 
changed fields since responding (1). Most respondents, however, reported 
"don’t know" (5) rather than the reasons we expected. We thought 
respondents might have problems deciding whether to report the population to 
which they teach (such as the gifted or mentally retarded) or the subject matter 
they teach (such as English or Music). (One respondent gave that reason.) 

We also thought respondents might have problems distinguishing between 
some of the answer categories, such as basic skills and remedial education 
versus special education. (No one gave that reason.) 



Table 12. Question 5b (Yes/No Response) - 562 Responses 



No. of 


Before Reconciliation 


After Reconciliation 


Categories 


GDR% 


GDR% | 


2 


3.7 


3.0 1 




(2.4, 5.1) 


(1.8, 4.2) | 
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Table 13. Question 5b (Field Code) - 117 Responses 



Part 1. Gross Difference Rates and Confidence limits (%) 


No. of 








Categories 


Before Reconciliation 


After Reconciliation 


7 


3.4 


no responses change with 




(0.7, 6.2) 


reconciliation 




Part 2. Significant NDRs - After Reconciliation 


Response Category 


NDR% 


Special areas 




-3.4 






(-7.6, 


-0.4) 


Part 3. Reasons for Difference between Responses 


Reason 


Count 


Percent 


Total 




10 


100.0 


Don’t know 




5 


50.0 


Teach both assignments 


3 


30.0 


Other assignment changed since responding 


1 


10.0 


Wasn’t sure whether to report population/subject 


1 


10.0 



Question 6a. Do you have a teaching certificate in this State in this MAIN 
teaching assignment field? 

This question has two reconciliation questions for determining the reasons for 
response differences. Reconciliation question 1 applied when the respondent 
answered "no” in the original interview and "yes” in the reinterview. 
Reconciliation question 2 applied when the reverse situation existed. We 
combined the reasons from the two reconciliation questions in Appendix D. 



Table 14. Question 6a - 616 Responses 



No. of 


Before Reconciliation 


After Reconciliation 


Categories 


GDR% 


GDR% | 


2 


1.5 


no responses change with 1 




(0.7, 2.3) 


reconciliation | 
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Question 6b. What type of certificate do you hold in this field? 

The NDR (1.5%) in part 2 of Table 15 reveals that respondents tended to 
over-report having an advanced professional certificate in the original 
interview. Part 3 of Table 15 shows that 40.6 percent (13) of the respondents 
who gave a reason for inconsistent answers said they had problems 
understanding the answer categories. Approximately 15 percent (5) said that 
none of the choices adequately described their certificate, and 12 percent (4) 
said that they didn’t know what kind of certification they had. 



Table 15. Question 6b - 523 Responses 



Part 1. Gross Difference Rates and Confidence Limits (%) 


No. of 








Categories 


Before Reconciliation 


After Reconciliation 


4 


6.3 


5.4 




(4.6, 8.1) 


(3.7, 7.0) 


Part 2. Significant NDRs - After Reconciliation 


Response Category 


NDR% 


Advanced professional certificate 


1.5 






(0.2, 2.8) 


Part 3. Reasons for Difference between Responses 


Reason 


Count 


Percent 


Total 




32 


100.0 


Wasn’t sure what one or more categories meant 


13 


40.6 


No choice described certification 


5 


15.6 


Don’t know what certification they have 


4 


12.5 


Misunderstood question 


2 


6.3 


Forgot/remembered information 


2 


6.3 


Other 




2 


6.3 


Received certificate since responding 


1 


3.1 


Missed skip pattem/question 


1 


3.1 


Refused to answer in one interview 


1 


3.1 


Don’t know 




1 


3.1 
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Question 6c. Was this certificate granted within the last 12 months? 



Table 16. Question 6c - 451 Response 



No. of 


Before Reconciliation 


After Reconciliation 


Categories 


GDR% 


GDR% 


2 


3.8 


no responses change 




(2.3, 5.2) 


with reconciliation 



Question 7a. Do you have a teaching certificate in this state in your 
OTHER teaching assignment field at this school? 

Similar to Question 6a, this question has two reconciliation questions. 
Reconciliation question 1 applied when the respondent answered "no" in the 
original interview and "yes" in the reinterview. Reconciliation question 2 
applied when the reverse situation existed. We combined the reasons from the 
two reconciliation questions in Table 17a and in Appendix D. 

The before reconciliation GDR (13.4 percent) in part 1 of Table 17a is one of 
the largest obtained for this questionnaire, suggesting that respondents had 
problems answering this question consistently. In part 3 of Table 17a, sixty 
percent (22) of the respondents who gave a reason for inconsistent answers 
said they either didn’t know why their answers differed or they misunderstood 
the question. The NDRs, however, suggest the problem lies with the "no" and 
"not applicable” answer categories. The NDRs for these categories in part 2 
of Table 17a are 5.4 and -6.6, respectively. They indicate that respondents 
had a tendency to overstate or say "no" in the original interview in lieu of "not 
applicable." 

Originally, we thought that this finding was due to mode differences. We 
hypothesized that mail respondents marked "no” in response to this question 
because they thought it sufficed. We reasoned that they never came across the 
more appropriate answer, "not applicable." For respondents who answered 
"no" to a previous question, ”5b. Do you teach classes in OTHER fields at 
this school?", we figured that the reinterviewers, trained to read through all 
the answer categories, recognized that "not applicable" was correct and 
conveyed this to respondents. This would explain why respondents were 
unable to verbalize their mistake during the reconciliation. Their mistake was 
due to not perceiving an answer category rather than not understanding it. 

A review of edit change rates from the 1991 SASS seemed to confirm this 
hypothesis (Jenkins, 1992a). The edit change rate is the percentage of eligible 




31 



23 



cases whose answers changed because of inconsistencies in the data. The edit 
change rate for this question was high (35%). Presumably, respondents’ 
answers changed from "no" to "not applicable" during the edit because they 
responded "no" to the earlier question, "Do you teach classes in OTHER fields 
at this school?”. 

Also, Bates and DeMaio (1990) learned through cognitive research with this 
question that respondents marked "no" because they didn’t realize there was a 
better choice, "not applicable." 

Therefore, we looked at the reinterview data by mode to see if our hypothesis 
was directly substantiated. A closer inspection does indeed show that the vast 
majority of respondents (42) systematically reported "no" in the original 
interview and "not applicable" in the reinterview. These data are presented in 
Table 17b. The other five response combinations only contain from 4 to 11 
cases. However, the data do not substantiate that this mistake was more likely 
to occur without an interviewer as with one. Table 17c shows that forty-seven 
percent (18 of 38) of the self-administered respondents reported "no" then "not 
applicable" compared to sixty-four percent (21 of 33) of the interviewer- 
administered respondents. Neither of these percentages are significantly 
different from their expected values at the 90 percent confidence level. 
Interviewers seem to be administering the question in a solidly standardized 
fashion, and for some unknown reason respondents are systematically 
interpreting the question differently in the two interviews. 
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Table 17a. Question 7a - 595 Responses 



Part 1. Gross Difference Rates and Confidence Limits (%) 


No. of 








Categories 


Before Reconciliation 


After Reconciliation 


3 


13.4 


13.1 




(11.1, 15.7) 


(10.8, 15.4) 


Part 2. Significant NDRs - After Reconciliation 


Response Category 


NDR% 


No 




5.4 






(3.2, 7.6) 


Not applicable 




-6.6 






(-8.8, -4.3) 


Part 3. Reasons for Difference between Responses 


Reason 


Count 


Percent 


Total 




36 


100.0 


Don’t know 




17 


47.2 


Misunderstood question 


5 


13.9 


Missed skip pattem/question 


3 


8.3 


Other 




3 


8.3 


Realized certificate is in other field 


2 


5.6 


Realized certificate applies to another state 


2 


5.6 


Realized certificate is not in other field 


2 


5.6 


Forgot/remembered info 


1 


2.8 


FR error 




1 


2.8 
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Table 17b. Original and Reinterview R esponses to Question 7a. 
Interview-Reinterview Response Count 



Percent 



Total 

Yes-No 

Yes-NA 

No-Yes 

No-NA 

NA-Yes 

NA-No 



80 


100.0 


6 


7.5 


11 


13.8 


6 


7.5 


42 


52.5 


4 


5.0 


11 


13.8 



Table 17c. Percent of No-NA Responses by Original Interview’s Mode of 



Administration 





Mode of Administration 


Interview-Reinterview Response 


Mail Telephone Total N 


No-NA 


18 21 39 


All other combinations 


20 12 32 


Total N 


38 33 71 1 



X 2 =1.87 d.f. = l p>.10 

1 Nine cases were treated as missing because they could not be identified by mode. 



Question 7b. What type of certificate do you hold in this field? 



Table 18. Question 7b - 67 Responses 



No. of 


Before Reconciliation 


After Reconciliation 


Categories 


GDR% 


GDR% 


4 


9.0 


no responses change 




(3.2, 14.7) 


with reconciliation 
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Question 7c. Was this certificate granted within the last 12 months? 
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Table 19. Question 7c - 61 Responses 



No. of 


Before Reconciliation 


After Reconciliation 


1 Categories 


GDR% 


GDR% 1 


1 2 


1.6 


no responses change 


1 


(-1.0, 4.3) 


with reconciliation | 



Question 8. In what grade levels are the students in your classes at THIS 
school? 

For this question, the respondents marked all answer categories that applied. 
For our analysis, we considered each of the 16 answer categories as a separate 
question with two possible answer categories: marked and unmarked. 

Although not statistically significant, the GDRs given in part 1 of Table 20 
appear to exhibit a slight curvilinear relationship. The GDRs tend to increase 
until they reach a maximum at the 5th grade, after which they tend to 
decrease. Furthermore, part 2 of this same table suggests that respondents 
tended to overreport students in the 4th through 8th grades in the original 
interview. 

Respondents’ reasons for inconsistent answers given in part 3 shed some light 
on these results. 

• A third (16) of the respondents said they did not know why a difference 
occurred. 

• However, another third (15) reported misunderstanding some aspect of 
the question. Specifically, they reported misunderstanding what was 
meant by "grade level or class (4), or they weren’t certain if they 
should report either the grade levels of students they sometimes teach 
(4) or classes with only a few students (1). Or else, they simply 
reported misunderstanding the question (6). 

• Three respondents had difficulty because they taught special students. 
These respondents either had trouble reporting the equivalent grade 
levels for the students, or they were not certain whether they should 
report them as ungraded or in their equivalent graded levels. 



35 



27 

All of this suggests the intent of this question is not perfectly clear to 
respondents. 



Table 20. Question 8 - 629 Responses 



Part 1. Gross Difference Rates and Confidence Limits (%) 


Answer 


Before Reconciliation 


After Reconciliation 


Category 


GDR% 


Limits 


GDR% Limits 


Ungraded 


0.2 


(-0.1, 0.4) 


no responses change 


Prekindergarten 


0.6 


(0.1, 1.2) 


no responses change 


Kindergarten 


1.9 


(1.0, 2.8) 


no responses change 


1st 


2.5 


(1.5, 3.6) 


2.4 (1.4, 3.4) 


2nd 


3.0 


(1.9, 4.1) 


no responses change 


3rd 


2.5 


(1.5, 3.6) 


no responses change 


4th 


2.9 


(1.8, 4.0) 


no responses change 


5th 


3.2 


(2.0, 4.3) 


2.9 (1.8, 4.0) 


6th 


1.9 


(1.0, 2.8) 


1.6 (0.8, 2.4) 


7th 


2.7 


(1.6, 3.8) 


2.4 (1.4, 3.4) 


8th 


2.7 


(1.6, 3.8) 


2.4 (1.4, 3.4) 


9th 


2.5 


(1.5, 3.6) 


2.2 (1.3, 3.2) 


10th 


2.1 


(1.1, 3.0) 


no responses change 


11th 


1.7 


(0.9, 2.6) 


no responses change 


12th 


1.9 


(1.0, 2.8) 


no responses change 


Postsecondary 


0.5 


(0.0, 0.9) 


no responses change 
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Part 2. Significant NDRs (%) - After Reconciliation 



Answer 

Category 


Marked 

NDR% Limits 


Not Marked 
NDR% Limits 


4th 


1.3 (0.1, 2.4) 


-1.3 (-2.4, -0.1) 


5th 


1.6 (0.5, 2.7) 


-1.6 (-2.7, -0.5) 


6th 


1.0 (0.1, 1.8) 


-1.0 (-1.8, -0.1) 


7th 


1.4 (0.4, 2.5) 


-1.4 (-2.5, -0.4) 


8th 


1.7 (0.7, 2.8) 


-1.7 (-2.8, -0.7) 



Part 3. Reasons for Difference between Responses 



Reason 



Total 

Don’t know 
Misunderstood question 
Teaching different students since responding 
Unsure whether to report grade levels of students 
sometimes taught 

Misunderstood what "grade level/class" meant 
Forgot/remembered info 
FR error 

Teach special students - difficulty reporting equivalent 
graded levels 
Other 

Misunderstood reference period 
Teach special students - unsure whether to report as 
"ungraded" or equivalent graded levels 
Unsure whether to report level of classes with few 
students 



Count 



Percent 



49 


100.0 


16 


32.7 


6 


12.2 


4 


8.2 


4 


8.2 


4 


8.2 


4 


8.2 


3 


6.1 


2 


4.1 


2 


4.1 


2 


4.1 


1 


2.0 


1 


2.0 
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Question 9. Which of the following best describes the community in which 
this school is located? 

This question has a misprint on the reinterview questionnaire. Answer 
category 6 should read "A suburb of a large city." Instead, it reads "A suburb 
of a very large city," the same as answer category 8. This misprint might 
have confused reinterviewers. Five respondents changed between answer 
categories 6 and 8 from the original to the reinterview. 

The answer categories for this question gave respondents trouble. As shown 
in part 3 of Table 21, approximately one-third (14) of the respondents who 
gave a reason for inconsistent answers said that they weren’t sure of the 
population of their school’s community. One-fourth (11) said that more than 
one answer category applied. These results are consistent with those of 
cognitive research on this item (Jenkins and DeMaio, 1990). 

This question was dropped from the SASS subsequent to the TFS reinterview. 
However, the results of the reinterview support this decision. 



Table 21. Question 9 - 617 Responses 



Part 1. Gross Difference Rates and Confidence Limits (%) 


No. of 








Categories 


Before Reconciliation 


After Reconciliation 


11 


8.3 


7.5 




(6.4, 10.1) 


(5.7, 9.2) 


Part 3. Reasons for Difference between Responses 


Reason 


Count 


Percent 


Total 




43 


100.0 


Wasn’t sure of population of community 


14 


32.6 


More than one category describes 






community 




11 


25.6 


Don’t know 




9 


20.9 


Misunderstood question 


4 


9.3 


FR error 




2 


4.7 


Other 




2 


4.7 


Missed skip pattem/question 


1 


2.3 
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Question 10. Are you currently teaching in the SAME school as you were 
last year when you completed the Schools and Staffing Survey? 



Table 22. Question 10 - 619 Responses 



No. of 
Categories 


Before Reconciliation 
GDR% 


After Reconciliation 
GDR% 


2 


1.6 

(0.8, 2.4) 


1.5 

(0.7, 2.2) 



Question 11. Are you currently teaching in the same state as you were last 
year when you completed the Schools and Staffing Survey? 

If respondents answered "no" to this question, they were asked a follow-up 
question: In which state or country are you now teaching? To answer this 
question, they supplied a two-letter abbreviation in the box provided and 
wrote-in the name of the state or country on the line provided. 



Table 23. Question 11 (Yes/No Response) - 179 Responses 



No. of 


Before Reconciliation 


After Reconciliation 


Categories 


GDR% 


GDR% 


2 


0.0 


no responses change 




(0.0, 0.0) 


with reconciliation 



Table 24. Question 11 (State Code) • 33 Responses 



No. of 
Categories 


Before Reconciliation 
Disagreement Rate (%) 


After Reconciliation 
Disagreement Rate(%) | 


State Code 
(2) 


6.1 

(-0.8, 12.9) 


no responses change with 
reconciliation | 
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Question 12. Which of the following best describes your move from last 
year’s school to this year’s school? 



Table 25. Question 12 - 179 Responses 



No. of 
Categories 


Before Reconciliation 
GDR% 


After Reconciliation 
GDR% 


5 


2.2 

(0.4, 4.1) 


no responses change 
with reconciliation 


Que< 

with 

non- 

Table 26. Quest 


;tion 13. Is the private school in which j 
the Roman Catholic Church or another 
religious? 

ion 13 - 33 Responses 


rou currently teach affiliated 
religious organization, or is it 


No. of 
Categories 


Before Reconciliation 
GDR% 


After Reconciliation 
GDR% 


3 


0.0 

(0.0, 0.0) 


no responses change 
with reconciliation 



Questions 29. b. (1-3) - Earnings 

Questions 29b(l) through 29b(3) all refer to before-tax earnings from teaching 

and other employment from the summer of 1991 through the end of the 1991- 

92 school year. 

• Question 29b(l) requires a monetary response only. According to 
reinterview instructions, the dollar values disagree if they exceed a 
$1,000.00 difference. 

• Question 29b(2) requires two responses: a yes/no response and a 
monetary response if respondents answer "Yes" to "additional 
compensation for extra curricular or addition activities.” According to 
reinterview instructions, the dollar values disagree if they exceed a 
$100.00 difference. 

• Question 29b(3) also requires two responses: a yes/no response and a 
monetary response if respondents answer "Yes" to "additional 
compensation from outside school system." According to reinterview 
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instructions, the dollar values disagree if they exceed a $100.00 
difference. 

Respondents had difficulty reporting monetary values in all three questions. 
The before reconciliation disagreement rates in part 1 of Tables 27, 29 and 31 
show this. They range from approximately 14 to 25 percent. The 
predominant reason for monetary differences is that respondents were unsure 
of the exact amount of their earnings. Part 3 of Tables 27, 29 and 31 show 
that over half of the respondents who gave a reason for inconsistent answers 
offered this as their reason. This suggests that respondents do not have easily 
accessible, precise figures stored in memory to accurately answer the monetary 
questions. It also suggests an inability or unwillingness on the respondent’s 
part to look up appropriate records which may exist. 

Question 29b(l). What is your academic base year salary for teaching in 
this school? 



Table 27. Question 29b(l) - 629 Responses 



Part 1. Disagreement Rates and Confidence Limits (%) 


No. of 


Before Reconciliation 


After Reconciliation 


Categories 








$ Value 


14.8 


10.5 


(2) 


(12.5, 17.1) 


(8.5, 12.5) 


Part 3. Reasons for Difference between Responses 


Reason 


Count 


Percent 


Total 




109 


100.0 


Unsure of exact amount 


71 


65.1 


Salary changed since responding 


9 


8.3 


Don’t know 




9 


8.3 


Included other salary earnings 


4 


3.7 


Misunderstood question 


3 


2.8 


FR error 




3 


2.8 


Included another source of income 


2 


1.8 


Forgot/remembered info 


2 


1.8 


Misunderstood reference period 


2 


1.8 


Manual/general error 


2 


1.8 


Wasn’t sure how to report as an itinerant teacher 


1 


0.9 


Gave after-tax earnings 


1 


0.9 



0 

ERIC 
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Question 29b (2). Do you, or will you earn any additional compensation 
from your school for extra curricular or additional activities such as 
coaching, student activity sponsorship, or evening classes? If yes, how 
much? 

For the yes/no portion of this question, approximately one-third (14) of the 
respondents who gave a reason for inconsistent answers said their activities 
changed since responding. This is a legitimate excuse and does not reflect a 
questionnaire problem. 

However, both the yes/no and the monetary portions of this question suggest 
that respondents’ memories do contribute to response differences. Part 3 of 
Table 28 shows that 20 percent (9) of the respondents who gave a reason for 
yes/no differences said they either remembered or forgot activities between the 
original interview and the reinterview. Likewise, in part 3 of Table 29, 25 
percent (6) of the respondents gave this reason for monetary differences. 

Still, as mentioned above for all the earnings questions, the leading reason for 
monetary differences is that respondents were unsure of the exact amount of 
their earnings. 
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Table 28. Question 29b(2) (Yes/No Response) - 588 Responses 



lAUIW — — v— / \ m. 

Part 1. Gross Difference Rates and Confidence limits ( c / 


SO 


No. of 
Categories 


Before Reconciliation 


After Reconciliation 


3 


4.6 

(3.2, 6.0) 


3.7 

(2.5, 5.0) 


Part 3. Reasons 


for Difference between Responses 


Reason 1 


Count { 


Percent 


Total 

Taken on/given up activities since responding 
Remembered/forgot activities _ 

Don’t know 

Unsure what "additional compensation" meant 
Other 

Missed skip pattem/question 
Changed jobs since responding 
Misunderstood question 
Manual/general error 
FR error 

Didn’t know what to include/exclude 


45 

14 

9 

8 

5 

2 

2 

1 

1 

1 

1 

1 


100.0 

31.1 

20.2 
17.8 
11.1 

4.4 

4.4 

2.2 

2.2 

2.2 

2.2 

2.2 
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Table 29. Question 29b(2) (Monetary Response) - 167 Responses 



Part 1. Disagreement Rates and Confidence Limits (%) 


No. of 








Categories 


Before Reconciliation 


After Reconciliation 


$ Value 


20.4 


18.6 


(2) 


(15.2, 25.5) 


(15.2, 25.5) 


Part 3. Reasons 


for Difference between Responses 


Reason 


Count 


Percent 


Total 




24 


100.0 


Wasn’t sure of exact amount 


13 


54.2 


Remembered/forgot activities 


6 


25.0 


Reported expected earnings 


2 


8.3 


Don’t know 




1 


4.2 


Unsure what "additional compensation meant 


1 


4.2 


Misunderstood question 


1 


4.2 



Question 29b(3). Do you, or will you, earn additional compensation from 
working in any job outside the school system? If yes, how much? 

For the yes/no response, the NDRs in part 2 of Table 30 reveal that 
respondents tended to underreport earning additional compensation from jobs 
outside the school system in the first interview. One explanation for this could 
be that the act of answering the first interview prompted respondents to 
remember additional compensation in the second interview. However, only a 
couple of respondents offered this reason. Part 3 of Table 30 shows that 30 
percent (14) of the respondents said they either misunderstood the question, 
misunderstood the reference period or were unsure of what constituted a job 
outside of the school system. 

For the monetary response, Part 3 of Table 31 again shows that the leading 
reason for differences is that respondents were unsure of the exact amount of 
their earnings. 
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Table 30. Question 29b(3) (Yes/No Response) - 599 Responses 



Fart 1. Gross DifTerence Rates and Confidence limits (%) 


No. of 
Categories 


Before Reconciliation 


After Reconciliation 


3 


5.8 


5.5 




(4.3, 7.4) 


(4.0, 7.0) 


Part 2. Significant NDRs - After Reconciliation 






Response Category 


NDR% 


Yes 




- 2.7 


No 




(-4.2, -1.1) 
2.5 






(0.9, 4.1) 


Part 3. Reasons for Difference between Responses 





Reason 


Count 


Percent 


Total 


46 


100.0 


Don’t know 


13 


28.3 


Unsure what constituted job 


7 


15.2 


Took up job since responding 


4 


8.7 


Misunderstood question 


4 


8.7 


Missed skip pattem/question 


3 


6.5 


Misunderstood reference period 


3 


6.5 


Quit job since responding 


2 


4.3 


Other 


2 


4.3 


Forgot/remembered info 


2 


4.3 


FR error 


2 


4.3 


Didn’t know what to include/exclude 


2 


4.3 


Refused to answer in one interview 


1 


2.2 


Manual/general error 


1 


2.2 
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Table 31. Question 29b(3) (Monetary Response) - 105 Responses 



No of 
Categories 


Before Reconciliation 


After Reconciliation 


$ Value 
(2) 


25.7 

(18.7, 32.7) 


23.6 

(16.8, 30.4) 


Part 3. Reasons for Difference between Responses 





Reason 



Total 



Wasn’t sure of exact amount 
Reported expected earnings 
Don’t know 

Refused to answer in one interview 
Misunderstood reference period 



Count 



14 

8 

2 

2 

1 

1 



Percent 



100.0 

57.2 

14.2 

14.3 
7.1 
7.1 



Question 29b(3a). Which of these best describes this job outside the school 
system? 



Table 32. Question 29b(3a) - 111 Responses 



No. of 


Before Reconciliation 


After Reconciliation 


Categories 


GDR% 


GDR& | 


4 


5.4 


4.5 1 




(1.9, 8.9) 


(1.3, 7.7) | 



Question 30. Do you receive any income-in-kind in addition to or in lieu 
of your school salary? 

For this question, respondents marked all answer categories that applied. 
Twenty cases, or approximately three percent, have multiple responses. For 
our analysis, we deleted these cases and considered the question as a single- 
response question with eight possible answer categories. 

The results suggest that respondents had problems understanding what was 
meant by "income-in-kind." Part 3 of Table 33 shows that almost half (11) of 
the respondents who gave a reason for inconsistent answers said they were 
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unsure what to report as income-in-kind, including whether to report partial 
income-in-kind. Cognitive research substantiates this finding, as demonstrated 
by the following excerpt from an interview with a private school teacher 
(DeMaio, 1990): 

... [the respondent] said that he received a tuition break, not full 
tuition, for his children, so he wasn’t sure if he should mark "tuition 
for your children" or not. He decided he shouldn’t. 

Table 33. Question 30 - 586 Responses 



No. of 






Categories 


Before Reconciliation 


After Reconciliation 


8 


2.9 


2.6 




(1.8, 4.0) 


(1.5, 3.6) 


Part 3. Reasons for Difference between Responses 





Part 1. Gross Difference Rates and Confidence Limits 



Reason 


Count 


Percent 


Total 


23 


100.0 


Wasn’t sure what to report as income-in-kind 


7 


30.4 


Unsure whether to report partial income-in-kind 


4 


17.4 


Don’t know 


3 


13.0 


In come- in-kind changed since responding 


2 


8.7 


Missed skip pattem/question 


2 


8.7 


Forgot/remembered info 


2 


8.7 


Other 


1 


4.3 


Manual/general error 


1 


4.3 


Misread question 


1 


4.3 



Question 31. Which category represents the total combined income 
(include your own income) of ALL FAMILY MEMBERS age 14 and older 
in your household during 1991? Include money from jobs, net business or 
farm income, pensions, dividends, interest, rent, social security payments, 
and any other income received by family members in your household. 

The before reconciliation GDR (16.2 percent) in part 1 of Table 34 is the 
largest of any of the closed -ended questions on this questionnaire. It suggests 
that respondents had difficulty answering this question consistently. Part 3 of 
Table 34 shows that nearly half (41) of the respondents who gave a reason for 
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inconsistent answers said they were unsure of the exact amount. Again, this 
suggests that they do not have an easily accessible, precise figure stored in 
memory to accurately answer the question. A cross tabulation of inconsistent 
answers between the reinterview and original interview shows that respondents 
tended to choose answer categories that were next to each other in the two 
interviews. For instance, a respondent might choose the answer category 
$15,000*$19,000 in the original interview and $20,000-$24,000 in the 
reinterview, or vice versa. This information is summarized in Graph 1 . 

A group of reasons in Part 3 of Table 34, making up approximately 25 percent 
(21) of the total reasons, indicates that respondents also have difficulty 
completely understanding this question, TTiese reasons include not being sure 
what to include or exclude, misunderstanding the reference period, not being 
sure whether to include adult children and generally misunderstanding the 
question. 



Table 34. Question 31 - 604 Responses 



Part 1. Gross Difference Rates and Confidence Limits (%) 


No. of 








Categories 


Before Reconciliation 


After Reconciliation 


13 


16.2 


12.9 




(13.8, 18.7) 


(10.7, 15.2) 


Part 3. Reasons for Difference between Responses 






Reason 


Count 


Percent 


Total 




84 


100.0 


Unsure of exact amount 


41 


48.8 


Don’t know 




11 


13.1 


Unsure what to include/exclude 


8 


9.5 


Misunderstood reference period 


7 


8.3 


Wasn't sure whether to include adult children 


4 


4.8 


FR error 




4 


4.8 


Misunderstood question 


2 


2.4 


Refused to answer in one interview 


2 


2.4 


Other 




1 


1.2 


Missed skip pattem/question 


1 


1.2 


Forgot/remembered info 


1 


1.2 


Manual/general error 


1 


1.2 


Misread question 


I ill 


1 


1.2 
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Graph 1. Question 31 - 98 Response Differences 

Response Category Differential between the Original and Reinterview 
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B. Question-by-Question Review of the Former Teacher Follow-Up Survey 
(TFS-2) 

In this section we analyze the reinterview questions from the 1992 Former 
Teacher Follow-up Survey (TFS-2). We use the same table format and criteria 
for explanations as above. Again, all question numbers refer to the original 
(TFS-2), not the reinterview (TFS-2R) questionnaire. 

Appendix E offers an inclusive list of the after reconciliation NDRs for all 
answer categories by question. Appendix F offers an inclusive list of all the 
reasons for differences for each question. 



o 
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Cover Question - Are you currently teaching - full-time, part-time, or as a 
long-term substitute - in grades K through 12? 



Table 35. Cover Question - 683 Responses 



No. of 


Before Reconciliation 


After Reconciliation 


Categories 


GDR(%) 


GDR(%) 


2 


1.6 


1.5 




(0.8, 2.4) 


(0.7, 2.2) 



Question 1. What is your MAIN OCCUPATIONAL status? 

Part 3 of Table 36 shows that a third (1 1) of the respondents who gave a 
reason for inconsistent answers said their occupational status changed since 
responding to the TPS. This does not indicate a problem with the 
questionnaire. However, another third reported that the answer categories 
were confusing or that they did not fit their situations. The NDRs in Part 2 of 
Table 36 reveal that respondents tended to overstate the first and last answer 
categories in the original interview. The overstated categories are "working in 
a elementary or secondary school with assignment OTHER THAN teaching" 
(NDR=0.8%) and "other" (NDR=1.5%), respectively. 

The first answer category may be too long and complicated for respondents to 
understand. Perhaps they simplify it to fit their situation and see instead, "an 
assignment OTHER THAN teaching." The fact that respondents had a 
tendency to choose the "other" category in the first interview further 
demonstrates that they had trouble matching their situations to one of the 
forced choices, which is consistent with the reason they gave during 
reconciliation. 
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Table 36. Question 1 - 664 Responses 



Part 1. Gross Difference Rates and Confidence Limits (%) 


No. of 
Categories 


Before Reconciliation 


After Reconciliation 


7 


6.9 

(5.3, 8.5) 


6.2 

(4.6, 7.7) 


Part 2. Significant NDRs - After Reconciliation 


Response Category 


NDR% 


Working in elementary/secondary school with assignment 
OTHER THAN teaching 
Other 


08 1 
(0.1, 1.4) 

1.5 

(0.2, 2.8) 


Part 3. Reasons for Difference between Responses 


Reason 


Count 


Percent 



Total 

Occupational status changed since responding 

Category problems 

Misunderstood question 

Other 

FR error 

Forgot/remembered info 
Manual/general error 



33 

11 

11 

4 

3 

2 

1 

1 



100.0 

33.3 

33.3 

12.1 

9.1 

6.1 
3.0 
3.0 



Question 2. What is your MAIN school assignment? 



Table 37. Question 2-74 Responses 



No. of 


Before Reconciliation 


After Reconciliation 


Categories 


GDR(%) 


GDR(%) 


6 


2.7 


no responses change with 




(-0.4, 5.8) 


reconciliation 
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Question 3a. For whom do you work? 

Since this question required a write-in response, we did not compute any 
mea s ures for it. However, we do include the reasons for the difference 
between the reinterview and original responses in Appendix F. 

Question 4. Which of these best describes your position as an 
EMPLOYEE? 



Table 38. Question 4 - 185 Responses 



No. of 
Categories 


Before Reconciliation 
GDR(%) 


After Reconciliation 
GDR(%) 


5 


1.6 

(0.1, 3.1) 


no responses change with 
reconciliation 



Question 5. Altogether, how much do you usually earn at this job before 
deductions? 

This question requires two responses: a monetary and a pay period response. 
The disagreement rate for the monetary response includes in its base only 
those cases where the pay periods agree. According to reinterview 
instructions, the dollar values disagree if the pay periods are the same and the 
dollar values differ by 

• S 1 .00 or more per hour 

• $5.00 or more per day 

• $20.00 or more per week 

• $50.00 or more biweekly 

• $100.00 or more per month 

• $1,000.00 or more per year. 

Reinterviewers reconciled responses if the dollar values disagreed or if the pay 
periods differed. 

The results here are similar to those for the income questions (29b(l-3)) from 
the Current Teacher Follow-up Survey (TFS-3). Respondents had difficulty 
reporting their earnings consistently. Table 39 shows that 21 percent (31) of 
the respondents gave different monetary responses for the same pay period. 
Part 3 of Table 40 shows that 35 percent (18) of the respondents who gave a 
reason for inconsistent answers said they weren’t sure of the exact amount. 
Another 15 percent (8) said they weren’t sure what to report as earnings. 
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Table 39. Question 5 (Monetary Response) - 151 Responses 



No. of 
Categories 


Before Reconciliation 
Disagreement Rate (%) 


After Reconciliation 
Disagreement Rate(%) 


$ Value 
(2) 


20.5 

(15.1, 25.9) 


17.5 

(12.5, 22.6) 




Part 1. Gross Difference Rates and Confidence Limits (%) 


No. of 








Categories 


Before Reconciliation 


After Reconciliation 


8 


9.6 


7.8 




(5.8, 13.3) 


(4.4, 11.2) 


Part 3. Reasons for Difference between Pay Rate Responses 


Reason 


Count 


Percent 


Total 




52 


100.0 


Wasn’t sure of exact amount 


18 


34.6 


Salary changed since responding 


8 


15.4 


Wasn’t sure what to report 


8 


15.4 


Reported in different units 


4 


7.7 


Changed jobs since responding 


3 


5.8 


Thought question asked for net earnings 


2 


3.8 


Missed skip pattem/question 


2 


3.8 


Other 




1 


1.9 


Don’t know 




1 


1.9 


Misunderstood question 


1 


1.9 


Refused to answer in one interview 


1 


1.9 


Forgot/remembered info 


1 


1.9 


Manual/general error 


1 


1.9 


FR error 




1 


1.9 
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Question 9. Have you earned any new degrees in the past 12 months? 

This question has two reconciliation questions. Reconciliation question 1 
applied when the respondent answered "no" in the original interview and "yes" 
in the reinterview. Reconciliation question 2 applied when the reverse 
situation existed. We combined the reasons from the two questions in 
Appendix F. 



Table 41. Question 9 - 657 Responses 



No. of 
Categories 


Before Reconciliation 
GDR(%) 


After Reconciliation 
GDR(%) 


2 


0.8 

(0.2, 1.3) 


no responses change 
with reconciliation 



Question 10. When did you earn this degree? 

This question requires two responses: month and year. The disagreement rate 
for the month response includes only those cases where the years agree. 



Table 42. Question 10 (Month Response) - 22 Responses 



| No. of 

| Categories 


Before Reconciliation 
Disagreement Rate (%) 


After Reconciliation 
Disagreement Rate(%) 


Month 

L_ w 


4.5 

(-2.8, 11.9) 


no responses change with 
reconciliation 



Table 43. Question 10 (Year Response) - 23 Responses 



No. of 


Before Reconciliation 


After Reconciliation 


Categories 


Disagreement Rate (%) 


Disagreement Rate(%) 


Year 


4.3 


no responses change with 


(2) 


(-2.6, 11.3) 


reconciliation 
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Question 11. What type of degree is it? 



Table 44. Question 11 - 22 Responses 



No. of 


Before Reconciliation 


After Reconciliation 


Categories 


GDR(%) 


GDR(%) 


6 


0.0 


no responses change 




(0.0, 0.0) 


with reconciliation 



Question 12. What is the major field of study for your NEW degree? 

To answer this question, the respondent chose from a list of 79 fields of study, 
which were divided into 9 subject areas. For our analysis, we collapsed the 
79 fields into the 9 subject areas. Attachment 2 lists the 79 fields by subject 
area. 

Only eight respondents answered this question. All of the eight original and 
reinterview responses agreed by subject area. However, the low response for 
this question compared to the 22 responses for the other rel a te d questions 
(Questions 10, 11 and 13) indicates that while respondents consistently 
answered this question, they were also less likely to answer it than the other 
questions. Two explanations for this exist. One is that they were reluctant to 
answer this question because it required considerably more work than the 
others. Here, respondents must manipulate a list of 74 specified items. The 
other questions (e.g., Have you earned any new degrees in the past 12 
months?) are easy in comparison, requiring them at the very most to match 
their answers to only a handful of specified choices. The other explanation is 
that respondents tried to answer this question, but did not succeed in matching 
their response to one of the 74 provided and eventually gave up. 



Table 45. Question 12 - 8 Responses 



No. of 
Categories 


Before Reconciliation 
GDR(%) 


After Reconciliation 
GDR(%) 


9 


0.0 

(0.0, 0.0) 


no responses change 
with reconciliation 
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Question 13. For what purpose did you earn this degree? 
Table 46. Question 13-22 Responses 



No. of 


Before Reconciliation 


After Reconciliation 


Categories 


GDR(%) 


GDR(%) 


6 


9.1 


no responses change 




(-1.0, 19.2) 


with reconciliation 



Question 14. Are you currently enrolled in a degree program? 
Table 47. Question 14 - 651 Responses 



No. of 
Categories 


Before Reconciliation 
GDR(%) 


After Reconciliation 
GDR(%) 


3 


2.0 

(1.1, 2.9) 


no responses change 
with reconciliation 


Quei 

Table 48. Quest 


ftion 15. What type of degree are you pursuing? 
ion 15 - 89 Responses 


No. of 
Categories 


Before Reconciliation 
GDR(%) 


After Reconciliation 
GDR(%) 


6 


3.4 

(0.2, 6.5) 


no responses change 
with reconciliation 



Question 16. What is the major field of study for the degree you are 
pursuing? 

As in Question 12, the respondent chose from the same 79 fields of study. 
Again, we grouped these fields into the same nine subject areas for analysis. 

The same response pattern is evident for this question as for question 12; that 
is, response to this question dropped off in comparison to related questions 
(Questions 15 and 17). As with question 12, respondents were either less 
motivated or less capable of answering this question than the related questions. 




rr ^ 

5o 
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Table 49. Question 16 > 51 Responses 



No. of 


Before Reconciliation 


After Reconciliation 


Categories 


GDR(%) 


GDR(%) 1 


9 


7.8 


5.9 -] 




(1.7, 14.0) 


(0.5, 11.3) 1 



Question 17. For what purpose are you pursuing this degree? 

Part 3 of Table 50 shows that almost half (5) of the respondents who gave a 
reason for inconsistent answers said that both categories equally described their 
situation. Therefore, it may be unrealistic to expect respondents to reliably 
choose one of the answer categories over the others. Another half (5) said 
they had problems with the categories, indicating that the answer categories for 
this question may need to be defined better. 



Table 50. Question 17 - 89 Responses 



Part 1. Gross Difference Rates and Confidence Limits (%) 


No. of 








Categories 


Before Reconciliation 


After Reconciliation 


6 


12.4 


no responses changed with 




(6.6, 18.1) 


reconciliation 




Part 3. Reasons for Difference between Responses 




Reason 


Count 


Percent 


Total 




11 


100.0 


Both categories equally describe 






situation 




5 


45.5 


Category problems 


5 


45.5 


Don't know 


1 


9.1 



Question 21a. Do you have a "lifetime" teaching certificate? 

This question has two reconciliation questions for determining the reasons for 
the difference between the reinterview and original responses. Reconciliation 
question 1 applied when the respondent answered "no" in the original 
interview and "yes" in the reinterview. Reconciliation question 2 applied 
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when the reverse situation existed. We combined the reasons from the two 
questions in Table 51 and again in Appendix F. 

Part 3 of Table 51 shows that a quarter (5) of the respondents who gave a 
reason for inconsistent answers said they weren’t sure whether their certificate 
was a "lifetime" certificate. 



Table 51. Question 21a - 658 Responses 



Part 1. Gross Difference Rates and Confidence Limits (%) 


No. of 








Categories 


Before Reconciliation 


After Reconciliation 


3 


3.3 


no responses changed with 




(2.2, 4.5) 


reconciliation 




Part 3. Reasons for Difference between Responses 


Reason 


Count 


Percent 


Total 




20 


100.0 


Don’t know 




6 


30.0 


Unsure whether certification was lifetime 


5 


25.0 


Received certificate since responding 


2 


10.0 


Thought certificate was lifetime 


2 


10.0 


Other 




2 


10.0 


Wasn’t sure which state(s) question referred to 


1 


5.0 


Forgot/remembered info 


1 


5.0 


Misunderstood question 


1 


5.0 



Question 21b. Has there been a change in your teacher certification status 
since last year? 

For this question, respondents marked all answer categories that applied. 
However, only four cases have multiple responses. For our analysis, we 
deleted these four cases and considered the question as a single-response 
question with five possible answer categories. Three explanations may exist 
for the limited number of multiple responses. 

• One is that only four respondents really had a reason to mark more 
than one category. 




58 



50 



A second explanation is that it does not make sense to respondents to 
mark more than one category. 

A final explanation is that respondents overlooked the "Mark (X) all 
that apply." instruction. The TFS uses a question-on-the-left-answer- 
on-the-right format. Jenkins et al. (1992) conclude that respondents 
often did not read instructions below the question. This is because 
respondents generally began to search for the answer once they read the 
question. As a result, their thoughts and consequently, their eyes were 
drawn away from the left-hand side of the page, where the instruction 
lay, to the right-hand side, where they knew the answer categories to 
be. 



Part 3 of Table 52 shows that approximately half (8) of the respondents who 
gave a reason for inconsistent answers reported misunderstanding either the 
question or answer categories. 



Table 52. Question 21b - 388 Responses 



Part 1. Gross Difference Rates and Confidence Limits (%) 



No. of 
Categories 



Before Reconciliation 




Part 3. Reasons for Difference between Responses 



After Reconciliation 



3.6 

(2.1, 5.2) 



Reason 



Total 

Misunderstood question 
Change since responding 
Category problems 
Misunderstood reference period 
Didn’t know what to include/exclude 
Other 



Count 
15 

5 
4 
3 
1 
1 
1 



Percent 



100.0 

33.3 

26.7 

20.0 

6.7 

6.7 

6.7 



Question 28. Which category represents the total combined income 
(include your own income) of ALL FAMILY MEMBERS age 14 and older 
in your household during 1991? Include money from jobs, net business or 
farm income, pensions, payments, and any other income received by 
family members in your household. 
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Results here are very similar to those for Question 31 on the Current Teacher 
Follow-up Survey (TFS-3). Again, the before reconciliation GDR (22.9%) in 
part 1 of Table 53 is the largest of any closed-ended question on the TFS-2. 



Table 53. Question 28 - 643 Responses 



Part 1. Gross Difference Rates and Confidence limits (%) 


No. of 








Categories 


Before Reconciliation 


After Reconciliation 


13 


22.9 


19.3 




(20.1, 25.6) 


(16.7, 21.8) 


Part 2. Significant NDRs - After Reconciliation 


Response Category 


NDR% 


$20,000 - $24,000 


-1.2 






(-2.2, 


-0.3) 


$25,000 - $29,999 


2.0 






(0.8, 3.3) 


$75,000 - $99,999 


-1.1 






(-2.1, -0.1) 


Part 3. Reasons for Difference between Responses 


Reason 


Count 


Eercem 


Total 




136 


100.0 


Wasn’t sure of exact amount 


67 


49.3 


Wasn’t sure what to include/exclude 


25 


18.4 


Don’t know 




13 


9.6 


Misunderstood reference period 


11 


8.1 


Refused to answer question in one interview 


5 


3.7 


Manual/general error 


4 


2.9 


Misunderstood question 


3 


2.2 


Wasn’t sure whether to include adult children 


2 


1.5 


Other 




2 


1.5 


Forgot/remembered info 


1 


0.7 


Gave "usual" case answer 


1 


0.7 


FR error 




1 


0.7 


Category problems 


1 


0.7 
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Question 29. What is your current marital status? 



Table 54. Question 29 - 637 Responses 



No. of 


Before Reconciliation 


After Reconciliation 


Categories 


GDR(%) 


GDR(%) 


4 


0.8 


no responses change 




(0.2, 1.4) 


with reconciliation 



Question 30. How many children do you have who are dependent on you 
(and your spouse) for more than half of their financial support? 

This question requires respondents to write-in a number or to check the "none" 
box. For analysis, we constructed seven distinct categories. 

• No children • 4 children 

• 1 child • 5 children 

• 2 children • More than five children. 

• 3 children 

For this question, a large number of the reasons for response differences are 
legitimate. Part 3 of Table 55 shows that approximately 41 percent (12) of the 
respondents who gave a reason for a difference said the number of their 
dependent children changed since responding. 
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Table 55. Question 30 - 659 Responses 



No. of 
Categories 


Before Reconciliation 


After Reconciliation 


7 


4.7 

(3.3, 6.1) 


4.1 

(2.8, 5.4) 


Part 3. Reasons 


for Difference between Responses 





Part 1. Gross Difference Rates and Confidence limits (%) 



Reason 


Count 


Percent 


Total 


29 


100.0 


Number changed since responding 


12 


41.4 


Don’t know 


5 


17.2 


Wasn’t sure whether support was more than half 


4 


13.8 


Other 


3 


10.3 


Unsure what "children" meant 


3 


10.3 


Misunderstood question 


2 


6.9 



Question 31. What was the age of your youngest child on his/her last . 
birthday? 

This question requires respondents to write-in a number. For analysis, we 
grouped these entries into ten distinct categories. 



• None 

• 1 to 5 years 

• 6 to 10 years 

• 11 to 15 years 

• 16 to 20 years 



21 to 25 years 
26 to 30 years 
31 to 35 years 
36 to 40 years 
Other 



Similar to question 30, most of the reasons for response differences are 
legitimate. Part 3 of Table 56 shows that 71 percent (22) of the respondents 
who gave a reason for a difference said their child had a birthday since 
responding. 
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Table 56. Question 31 - 291 Responses 



Part 1. Gross Difference Rates and Confidence Limits (%) 


No. of 








Categories 


Before Reconciliation 


After Reconciliation 


10 


4.8 


no responses changed 




(2.7, 6.9) 


with reconciliation 


Part 3. Reasons for Difference between Responses 


Reason 


Count 


Percent 


Total 




31 


100.0 


Child had birthday since responding 


22 


71.0 


Misunderstood question 


3 


9.7 


Manual/general error 


3 


9.7 


Missed skip pattem/question 


1 


3.2 


FR error 




1 


3.2 


Other 




1 


3.2 



Question 32a. Do you have persons other than your spouse and children 
who are dependent on you for more than half of their f inancial support? 

Part 3 of Table 57 shows that nearly half (4) of the respondents who gave a 
reason for inconsistent answers said they had different ideas of what it means 
to be dependent between the original interview and the reinterview. 
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Table 57. Question 32a- 657 Responses 



Part 1. Gross Difference Rates and Confidence limits (5 


50 


No. of 
Categories 


Before Reconciliation 


After Reconciliation 


3 


1.7 

(0.9, 2.5) 


1.5 

(0.7, 2.3) 


Part 3. Reasons 


for Difference between Responses 






Reason 


Count 


Percent 


Total 

Had different ideas what dependent meant 
Unsure what "other persons" meant 
Situation changed since responding 
Other 

Wasn’t sure whether support was more than half 


9 

4 

2 

1 

1 

1 


100.0 

44.4 

22.2 

11.1 

11.1 

11.1 



Question 32b. How many persons other than your spouse or children are 
dependent on you for more than half of their financial support? 

Similar to Question 30, this question requires respondents to write-in a 
number. For analysis, we made four distinct categories. 

• None - that is, no persons 

• 1 person 

• 2 persons 

• Other - that is, more than two persons. 



Table 58. Question 32b - 18 Responses 



Part 1. Gross Difference Rates and Confidence limits (%) I 


No. of 
Categories 


Before Reconciliation 


After Reconciliation | 


4 


5.6 

(-3.3, 14.4) 


5.3 

(-3.2, 13.7) | 




6 



r 



56 



IV. Detailed Recommendations 

A. Proposed Questionnaire Revisions for Current Teacher Follow-up Survey 
(TFS-3) 

Question 1. MAIN activity and OTHER main activity? 

First, list the activities before the question series. This way all of the 
questions can refer back to one list. Second, break the original question into 
die following logical sequence of five simpler, more natural sounding 
questions. This sequence avoids the use of the awkwardly phrased "Other 
Main Activity." It eliminates respondents early on whose time is equally 
divided between two activities, thereby eliminating the need to maintain an 
"equal" characterization of the activities in the remaining questions. 

LIST OF ACTIVITIES 

1. Teaching in an elementary or secondary school 

2. Working in an elementary or secondary school with an assignment 
OTHER THAN teaching 

3. Working in an occupation outside of elementary or secondary 
education 

4. Attending a college or university 

5. Caring for family members 

6. Retired 

7. Other - Specify 



la. Is your time equally divided between two of the above listed 
activities? 

yes— skip to lb 

no— skip to 2 

lb. If so, what are their numbers from the list above? 

and — skip to 4 

2. What number from the list above best describes the activity you 
spend most of your time on during the work week; that is, what is 
your main activity? 

main activity 
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3a. Do you spend time on any other activity from the list above? 



yes— skip to 3b 

no— skip to 4 

3b. If so, what number from the list above best describes this other 
activity? 

other activity 



Question 3b 



Teaching Assi 



mi if hi 
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First, rearrange the answer categories. The itinerant and long-term substitute 
teachers are more likely to consider themselves regular full- or part-time 
teachers than vice versa. 



Secondly, reword the "itinerant teacher" answer category. State the definition 
of "itinerant teacher" first, then the technical term in parentheses, instead of 
vice versa. 



Finally, provide a more comprehensive list of familiar names for itinerant 
teachers, such as traveling, co-op, or satellite teachers. 



Our suggested order and wording are: 

You provide instruction at more than one school (i.e. f you are an 

itinerant, traveling, co-op, or satellite teacher). 

You fill the role of a regular teacher on a long-term basis, but you 

are still considered a substitute (i.e., you are a long-term substitute 
teacher). 

You are a regular full-time or part-time teacher. 



Question 4, Other Schoo l Assignment? 

Break this question into its constituent parts, thereby eliminating the need for 
the "not applicable" answer category. Also, change the phrase "employee 
with less than a full-time teaching assignment" to "part-time teacher," which is 
easier to understand. Our suggested wording is: 



ERIC 
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Are you a full-time school or district employee who is a part-time 
teacher? 

yes— skip to 4b 

no-skip to 5 

4b. Which of these best describes your non-teaching assignment at this 
school? 



Questions 5a-7c. MAIN and OTHER T eaching Assi gnments Fields? 

Rearrange the "teaching assignment” questions so that respondents are first 
a series of questions about their MAIN teaching assignment field, 
followed by a series of questions about their OTHER teaching assignment 
field. This will streamline the process for respondents who teach only one 
subject, allowing them to skip over the entire series of questions relating to 
their OTHER teaching assignment field. Jenkins made this recommendation 
once before (1992a, p.24), and the results of the TFS Reinterview support it. 

Presently, the questions alternate between asking teachers about their MAIN 
and OTHER teaching assignment fields. Respondents who teach two subjects 
must pay particular attention to which subjects they are being asked about. 
Respondents who teach only one subject are continually and alternately asked 
questions that don’t pertain to them. Our rearrangement is: 



Question 5a. 
Question 6a. 
Question 6b. 
Question 6c. 
Question 5b. 
Question 7a. 
Question 7b. 
Question 7c. 



MAIN teaching assignment? 

Teaching certificate in MAIN field? 

Type of certificate? 

Certificate granted within last 12 months? 
Teach classes in OTHER assignment fields? 
Teaching certificate in OTHER field? 

Type of certificate? 

Certificate granted within last 12 months? 



Respondents who respond "no” to 5b will skip over 7a through 7c. 
Consequently, 7a will no longer need the "not applicable" answer category. 



o 
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Question 6b. Tvne of certificate? 

Reverse the first two answer categories. Respondents are more likely to have 
"regular or standard state certificates" and confuse these with "advanced 
professional certificates" than vice versa. Also, add an "other" and a "don’t 
know" answer category. 

Question 8. Gra de Levels of Students? 

If the intent of this question is to learn what the grade levels are of all the 
students that the teacher teaches, then one possibility for rewording this 
question is: 

In what grade levels are the students that you teach at THIS 
school? 

This wording eliminates the confusing word c lass , the definition of which 
gives respondents problems. Does a class need to meet regularly to be 
considered a class? Does it need to be a certain size before it qualifies as a 
class? Respondents are not certain of the answers to these questions. 

If, however, the intent of this question is to learn what the grade levels are of 
students in regularly scheduled classes of certain sizes, then this needs to be 
made clear to respondents. For example: 

In what grade levels are the students in your regularly scheduled 
riaw< (those with X or more students) at THIS school? 

Questions 29bfl-31 and 31. Income? 

The problem posed by the income questions— the fact that respondents who 
gave a reason for inconsistent answers overwhelmingly said they were 
providing estimates - does not appear simple to solve. Initially we thought that 
asking respondents either 1) to obtain records to accurately answer the income 
questions or 2) to stop and think about them more carefully might be possible 
solutions to this problem. However, this now appears to have been a naive 
perspective. A recent experimental treatment in which respondents were asked 
to use their records to report sources of income did not show substantial 
improvements in the quality of data obtained, while it lead to lower response 
rates and higher field costs (Marquis, 1993). 

The results of this research suggest that simply asking respondents to use their 
records will probably not work. We need to have a better understanding of 
respondents’ use of records before we will be able to properly guide this 



68 



process. Jenkins (1992b) concludes that respondents’ use of records is one of 
the most complex areas of questionnaire research to study, since it requires in- 
depth knowledge about respondents’ records as well as how they use those 
records. Very little is known about that process to date. Certainly this is an 
area in need of further research. 

Given that asking respondents to use their records may have a detrimental 
effect on the data in other ways (i.e., increase non-response), the question 
becomes just how much measurement error in the data can the sponsor 
tolerate. This is another area that needs to be explored. Does the amount of 
response variance in the data affect the estimates or inferences drawn? If so, 
will the benefit of decreased measurement error outweigh the cost of increased 
nonresponse error? Further research is needed to address these difficult 
issues. 



Question 30 . Income-in-Kind? 

Reword this question so that respondents understand they are supposed to 
include partial sources of income-in-kind. 

Including partial sources, do you receive any income-in-kind in 
addition to or in lieu of your school salary? 

Proposed Questionnaire Revisions for Former Teachers (TFS-2) 

Question 1. Main Occupational Status 

Further research is needed to reduce discrepant answers to this question. We 
need to learn why respondents are systematically marking the "other” category 
in the first interview. 



Question 12. Major Fiel d of Study for New 






£ 



Research is needed to determine if respondents are less motivated or less 
capable of answering this question than the related questions. 



Question 21a. Lifetime Teaching Certificate 

Provide a "Don’t know" answer category, since respondents do not know if 
their teaching certificate is "lifetime" or not. 
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Question 21b. Change in Teacher Certification Status 

Vertically align the question and response categories (for this as well as all of 
the questions on the questionnaire) to help respondents perceive the 
instructions that follow the question. 

Questions 5 and 28. Income 

Our recommendation is the same here as for Questions 29b(l-3) and 31 on the 
TFS-3. 

Question 28a. Other Dependent Persons 

Further research is needed to reduce discrepant answers to this question. 
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Attachment A 
December 27, 1993 



Teaching Assignment Fields by Category 
Questions Sa and Sb on the TFS-3 



General 
Prekindergarten 
Kindergarten 
General elementary 

Special areas 
American Indian studies 
Art 

Basic skills and remedial education 
Bilingual education 
Computer science 
Dance 

Drama/Theater 
English/Language arts 
English as a second language 
Gifted 

Home economics 

Journalism 

Mathematics 

Military science 

Music 

Philosophy 

Physical education, health 

Reading 

Religion 

Social studies/Social science 

Foreign language 

French 

German 

Latin 

Russian 

Spanish 

Other foreign language 



Science 

Biology/Life science 
Chemistry 

Geology/Earth science/Space science 
Physics 

General and all other science 

Vocational-technical education 

Accounting 

Agriculture 

Business, marketing 

Health occupations 

Industrial arts 

Trade and industry 

Technical 

Other vocational education 

Special education 
Special education, general 
Emotionally disturbed 
Mentally retarded 
Speech/Language impaired 
Deaf and hard-of-hearing 
Visually handicapped 
Orthopedically impaired 
Mildly handicapped 
Severely handicapped 
Specific learning disabilities 
Other special education 

All others 
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Major Fields of Study by Category 
Questions 12 and 16 on the TFS-2 



General Education 
Pre-elementary/early childhood 
elementary education 
Secondary education 

Subject Area Education 
Agricultural education 
Art education 
Bilingual education 
Business, commerce, and 
distributive education 
Crosscultural education 
English education 
English as a second language 
Foreign languages education 
Home economics education 
Indian education 

Industrial arts, vocational and technical, 

trade and industry education 

Mathematics education 

Music education 

Physical education/Health 

Reading education 

Religious education 

Science education 

Social studies/Social sciences 

Special Education 
Special education, general 
Emotionally disturbed 
Mentally retarded 
Speech/Language impaired 
Deaf and hard-of-hearing 
Visually handicapped Orthopedically 
impaired 

Mildly handicapped 
Severely handicapped 
Specific learning disabilities 
Other special education 



Other Education 
Curriculum and instruction 
Educational administration 
Educational psychology 
Counseling and guidance 
Other education 

General 

Agriculture/natural resources 

American Indian studies 

Other area and ethnic studies 

Architecture and environmental design 

Art, fine and applies 

Business and management 

Communication and journalism 

Computer/information sciences 

Drama, theater 

Engineering 

English 

General studies 

Health professions/occupations 
Home economics 
Humanities 
Law 

Library science 

Mathematics 

Military science 

Multi/interdisciplinary studies 

Music 

Philosophy 

Psychology 

Public affairs and services 
Religion, theology 

Foreign Languages 

French 
German . 

Latin 

Russian 

Spanish 

Other foreign languages 



Natural Sciences 
Biology/Life sciences 
Chemistry 

Geology/Earth science 
Physics 

Other natural sciences 
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Social Sciences 

Economics 

History 

Political science/govemment 
Sociology 

Other social sciences 



Other 



Appendix A-l 
December 27, 1993 



TFS-3 for Current Teachers 



Before Reconciliation L-fold Indexes 





Question 


L-fold 


Confidence 

Limits 


Cover. 


Still Teaching 


+ + 


mm 


1 . 


MAIN activity and OTHER main activity? 
Main activity 
Other activity 


19.6 

9.7 


(11.4, 33.8) 
(6.2, 14.9) 


2. 


Position as a school EMPLOYEE? 


7.9 


(4.3, 14.6) 


3a. 


Time as a TEACHER? 


7.8 


(4.5, 13.4) 


3b. 


Teaching assignment? 


22.2 


(13.9, 35.6) 


4. 


Other school assignment? 


16.8 


(10.9, 26.0) 


5a. 


MAIN teaching assignment at this school? 


2.3 


(1.4, 3.9) 


5b. 


Teach classes in OTHER fields at Uiis school? 
Yes/no response 


12.3 


(8.6, 17.5) 




Field code 


5.9 


(2.7, 13.2) 


6a. 


Teaching certificate in this State in this MAIN 
teaching assignment field? 


6.4 


(3.7, 10.9) 


6b 


Type of certificate? 


17.4 


(13.1, 23.1) 


6c. 


Certificate granted within the last 12 months? 


16.0 


(10.8,23.8) 


7a. 


Teaching certificate in this state in OTHER 
teaching assignment field? 


25.6 


(21.6, 30.4) 


7b. 


Type of certificate? 


** 


mm 


7c. 


Certificate granted within the last 12 months? 


mm 


mm 



A double asterisk (**) indicates a suppressed L-fold. This means there are too few cases to 
accurately compute the statistic. 
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== ■■■■■ 




Confidence 


Question 


L-fold 


Limits 


8. Grade levels of students in your classes at THIS 


. 




SQhpol? 






Ungraded 


** 


mm 


Prekindergarten 




mm 


, Kindergarten 


7.8 


(4.9, 12.5) 


1st 


8.1 


(5.4, 12.2) 


2nd 


10.1 


(6.9, 14.7) 


3rd 


8.2 


(5.4, 12.3) 


4th 


9.1 


(6.2, 13.4) 


5th 


9.6 


(6.7, 13.9) 


6th 


5.8 


(3.6, 9.3) 


7th 


7.7 


(5.2, 11.4) 


8th 


7.9 


(5.3, 11.7) 


9th 


6.6 


(4.4, 9.9) 


10th 


5.4 


(3.5, 8.5) 


11th 


4.6 


(2.8, 7.5) 


12th 


5.0 


(3.1, 8.0) 


Postsecondary 


** 


mm 


9. Community in which this school is located? 


9.9 


C7.9, 12.4) 


10. Teaching in the SAME school? 


3.9 


(2.3, 6.5) 


11. Teaching in same state? Yes/No response 


0.0 


(0.0, 5.0) 


12. Move from last year’s school to this year’s 






school? 


3.1 


(1.4, 6.8) 


13. Religious affiliation? 


mm 


mm 


29b(2). Additional compensation for extra 






curricular/additional activities from your 






school? Yes/No response 


10.8 


(7.9, 14.8) 


29b(3). Additional compensation from outside 






school system? Yes/No response 


17.9 


(13.6, 23.6) 


29b(3a). Job outside the school system? 


9.2 


(4.7, 17.8) 



A double asterisk (•*) indicates a suppressed NDR. This means there are too few cases to 
accurately compute the statistic. 
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Question 


L-fold 


Confidence 

Limits 


30. Income-in-kind? 


15.7 


(10.6, 23.4) 


31. Total combined income of ALL FAMILY 

MEMBERS age 14 and older in your household 
during 1991? 


18.2 


(15.7, 21.2) 



A double asterisk (**) indicates a suppressed NDR. This means there are too few cases to 
accurately compute the statistic. 
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TFS-2 for Former Teachers 
Before Reconciliation L-fold Indexes 



Question 


L-fold 


Confidence 

Limits 


Cover. 


Still Teaching? 


mm 


mm 


1 . 


MAIN OCCUPATIONAL status? 


8.6 


(6.8, 11.0) 


2. 


MAIN school assignment? 


4.4 


(1.5, 13.3) 


4. 


Position as an EMPLOYEE? 


5.5 


(2.2, 13.8) 


5. 


Altogether, how much do you usually earn at 
this job before deduction? 


14.7 


(9.8,22.1) 


9. 


New Degree? 


10.6 


(5.2, 21.8) 


11. 


What type of degree is it? 


mm 


mm 


12. 


Major field of study for NEW DEGREE? 


mm 


mm 


13. 


For what purpose did you earn this degree? 


mm 


mm 


14. 


Currently enrolled in a degree program? 


8.0 


(5.1, 12.5) 


15. 


What type of degree are you pursuing? 


6.2 


(2.5, 15.5) 


16. 


Major field of study for degree you are 
pursuing? 


mm 


mm 


17. 


For what purpose are you pursuing this degree? 


15.7 


(10.4, 25.0) 


21a. 


"Lifetime" teaching certificate? 


6.9 


(4.9, 9.8) 


21b. 


Change in teacher certification since last year? 


18.8 


(12.7, 28.0) 


28. 


Total combined income? 


25.4 


(22.6, 28.6) 


29. 


Current marital status? 


2.2 


(1.1, 4.5) 


30. 


Dependent children for more than half of their 
financial support? 


7.5 


(5.6, 10.0) 



A double asterisk (**) indicates a suppressed L-fold. This means there are too few cases to 
accurately compute the statistic. 
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Question 


L-fold 


Confidence 
Limits J 


31. Age of youngest child? 


5.9 


(3.8, 9.1) 1 


32a. Other dependent persons for more than half of 
their financial support? 


23.3 


(14.2, 38.0) 


32b. How many persons? 


mm 


mm 



Double asterisk (**) indicates a suppressed L-fold. This means there are too few cases to 
accurately compute the statistic. 
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TFS-3 for Current Teachers 
After Reconciliation NDRs by Response Categories 



Question/ 

Response Category 


NDR% 


Confidence 

Limits 


rover. Still Teaching? 

Yes 

No 


mm I 

mm I 




1. MAIN activity and OTHER main activity? Main 

activity 

Teaching in school 

Working in school other than teaching Working in 

occupation outside education 

Attending college/university 

Caring for family 

Retired 

Other 


-0.8* 
mm I 

mm 

mm I 

mm 

mm I 

mm 1 


(-1.6, -0.1) 


7 Position as a school EMPLOYEES 

Full-time employee 

> = 3/4 time, full-time 

> = 1/2 time, < 3/4 time 

> = 1/4 time, < 1/2 time 
< Less than 1/4 time 


0.2 

mm 

-0.2 

mm 

mm 


(-0.5, 0.8) 
(-0.7, 0.3) 


3a. Time as a TEACHER? 

Full-time teacher 

> = 3/4 time, full-time 

> = 1/2 time, < 3/4 time 

> = 1/4 time, < 1/2 time 
< Less than 1/4 time 


0.3 

mm 

-0.3 

mm 

mm 


(-0.4, 1.0) 
(-0.9, 0.3) 


3h Teaching assignment? 

Regular full /part-time teacher 
Itinerant teacher 
Long-term substitute 


1.6* 

-1.5* 

mm 


(0.7, 2.6) 
(-2.4, -0.6) 



A single asterisk (*) indicates a NDR significantly different from zero at the 90% confidence 
level. 



A double asterisk (**) indicates a suppressed NDR, or too few cases to accurately compute. 
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Question/ 

Response Category 


NDR% 


Confidence 

Limits 


4. Other school assignment? 






Administrator 


mm 




Counselor 


mm 


' 


Librarian/Media specialist 


mm 




Coach 


mm 




Other professional staff 


mm 




Support staff 


mm 




Other 


mm 




Not applicable 


0.0 


(-1.1, M) 


5a. MAIN teaching assignment at this school? 






General 


0.2 


(-0.5, 0.8) 


Special areas 


-0.2 


(-0.9, 0.6) 


Foreign language 


0.0 


(-0.4, 0.4) 


Science 


0.2 


(-0.2, 0.7) 


Vocational/technical education 


• • 


(-0.2, 0.7) 


Special education 


0.2 


All others 


• • 




5b. Teach classes in OTHER fields at this school? 
Yes/No response 






Yes 


-0.2 


(-1.4, 1.1) 


No 


0.2 


(-1.1. 1-4) 


5b. Teach classes in OTHER fields at this school? . 






Field code 






General 


mm 




Special areas 


- 3.4* 


(-7.6, -0.4) 


Foreign language 


mm 




Science 


mm 




Vocational/technical education 


mm 




Special education 


mm 




All others 

»■■ ! !!■ t 1 1 m 11 — 


mm 





A single asterisk (*) indicates a NDR significantly different from zero at the 90% confidence 
level. 



A double asterisk (**) indicates a suppressed NDR, or too few cases to accurately compute. 
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Question/ 
Response Category 


NDR% 


Confidence 

Limits 


6a. Teaching certificate in this State in this MAIN 
teaching assignment field? 

Yes 


0.8 


(-0.0, 1.6) 


No 


-0.8 


(-1.6, 0.0) 


6b. Tvpe of certificate? 






Advanced professional certificate 


1.5* 


(0.2, 2.8) 


Regular or standard state certification 


-1.1 


(-2.8, 0.5) 


Probationary certification 


mm 




Temporary, provisional or emergency 
certification 


-0.8 


(-1.7, 0.2) 


6c. Certificate granted within the last 12 months? 






Yes 


1.1 


(-0.4, 2.6) 


No 


-1.1 


(-2.6, 0.4) 


7a. Teaching certificate in this state in OTHER teaching 
assignment field? 

Yes 


1.2 


* (-0.2, 2.6) 


No 


5.4* 


(3.2, 7.6) 


Not applicable 


-6.6* 


(-8.8, -4.3) 


7b. Type of certificate? 






Advanced professional certificate 


mm 




Regular or standard state certification 


mm 




Probationary certification 


mm 




Temporary, provisional or emergency 
certification 


mm 





A single asterisk (*) indicates a NDR significantly different from zero at the 90% confidence 
level. 



A double asterisk (**) indicates a suppressed NDR, or too few cases to accurately compute. 
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Question/ 

Response Category 


NDR% 


Confidence 

Limits 


7c. Certificate granted within the last 12 months? 






Yes 


** 




No 


** 




8. Grade levels of students in your classes at THIS 






schwH 






Ungraded 


** 




Prekindergarten 


** 


(-0.9, 0.9) 


Kindergarten - marked 


0.0 


not marked 


0.0 


(-0.9, 0.9) 


1st - marked 


0.8 


(-0.2, 1.8) 


not marked 


-0.8 


(-1.8, 0.2) 


2nd - marked 


0.2 


(-1.0, 1.3) 


not marked 


-0.2 


(-1.3, 1.0) 


3rd - marked 


1.0 


(-0.1, 2.0) 


not marked 


-1.0 


(-2.0, 0.1) 


4th - marked 


1.3* 


(0.1, 2.4) 


not marked 


-1.3* 


(-2.4, -0.1) 


5th - marked 


1.6* 


(0.5, 2.7) 


not marked 


-1.6* 


(-2.7, -0.5) 


6th - marked 


1.0* 


(0.1, 1.8) 


not marked 


-1.0* 


(-1.8, -0.1) 


7th - marked 


1.4* 


(0.4, 2.5) 


not marked 


-1.4* 


(-2.5, -0.4) 


8th - marked 


1.7* 


(0.7, 2.8) 


not marked 


-1.7* 


(-2.8, -0.7) 


9th - marked 


0.6 


(-0.4, 1.6) 


not marked 


-0.6 


(-1.6, 0.4) 


10th - marked 


0.2 


(-0.8, 1.1) 


not marked 


-0.2 


(-1.1, 0.8) 


11th - marked 


0.2 


(-0.7, 1.1) 


not marked 


-0.2 


(-1.1, 0.7) 


12th - marked 


0.3 


(-0.6, 1.2) 


not marked 


-0.3 


(-1.2, 0.6) 


Postsecondary 


mm 





A single asterisk (*) indicates a NDR significantly different 



level. 



A double asterisk (**) indicates a suppressed NDR, or too few cases to accurately compute. 



O 
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Question/ 

Response Category 


NDR% 


Confidence 

Limits 


9. Community in which this school is located? 






Rural or farming community 


0.0 


(-1.0, 1.0) 


Small city or town 


-0.5 


(-1.7, 0.7) 


Medium-sized city 


0.8 


(-0.2, 1.9) 


Suburb of medium-sized city 


-0.2 


(-0.9, 0.6) 


Large city 


-0.3 


(-1.3, 0.6) 


Suburb of large city 


0.0 


(-1.0, 1.0) 


Very large city 


0.2 


(-0.5, 0.8) 


Suburb of very large city 


0.0 


(-0.7, 0.7) 


Military base or station 


** 




Indian reservation 


** 




Other 


mm 




10. Teaching in the SAME school? 






Yes 


-0.2 


(-1.0, 0.7) 


No 


0.2 


(-0.7, 1.0) 


11. Teaching in same state? Yes/No response 






Yes 


0.0 


(-1.5, 1.5) 


No 


0.0 


(-1.5, 1.5) 


12. Move from last year’s school to this year’s school? 






Moved from public to another public 




(-0.4, 3.4) 


school in same district 


1.1 


Moved from public school district 






to another public school district 


-1.7 


(-4.2, 0.1) 


Moved from private to public school 


1.1 


(-0.4, 3.4) 


Moved from private to another 






private school 


-0.6 


(-2.5, 0.6) 


Moved from public to private school 


** 





A single asterisk (*) indicates a NDR significantly different from zero at the 90% confidence 
level. 



A double asterisk (**) indicates a suppressed NDR, or too few cases to accurately compute. 
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Question/ 




Confidence 


Response Category 


NDR% 


Limits 


13. Religious affiliation? 






Religious - Roman Catholic 


mm 




Religious - Non-Roman Catholic 
Non-Religious 


WW 

mm 




29b(2). Additional compensation for extra 






curricular/additional activities from your 
school? 

Yes/No response 






Yes 


-0.2 


(-1.5, 1.1) 


No 


0.0 


(-1.3, 1.3) 


Don’t know/refused 


mm 




29b(3). Additional compensation from outside school 






system? Yes/No response 






Yes 


- 2.7* 


(-4-2, -1.1) 


No 


2.5* 


(0.9, 4.1) 


Don’t know/refused 


mm 




29bf3al. Job outside the school svstem? 






Teaching or tutoring 


-0.9 


(-3.7, 1.9) 


Nonteaching, but education related 


mm 


(-3.2, 3.2) 


Not related to education 


0.0 


Don’t know/refused 

m. T w — w-m - _ - ^ - 1.1- . ^ - — A 


mm 





level. 



A double asterisk (**) indicates a suppressed NDR, or too few cases to accurately compute. 
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Question/ 
Response Category 


NDR% 


Confidence 

Limits 


30. Income-in-kind? 






Housing or housing expenses 


mm 




Meals 






Tuition for your children 


mm 




Child care 


mm 




College tuition for yourself 


mm 




Car/tran spoliation expenses 


mm 




None of the above 


-0.2 


(-1.3, 0.9) 


Don’t know/refused 


mm 




31. Total combined income of ALL FAMILY 

MEMBERS age 14 and older in your household 
during 1991? 






Less than $10,000 


mm 




$ 10, 000-$ 14, 999 


mm 




$ 15,000-$19,999 


0.2 


(-0.2, 0.7) 


$ 20,000-$24,999 


-0.2 


(-0.8, 0.5) 


$ 25,000-$29,999 


-0.7 


(-1.6, 0.2) 


$ 30,000-$34,999 


-0.2 


(-1.5, 1.2) 


$ 35,000-$39,999 


0.8 


(-0.6, 2.2) 


$ 40,000-$49,999 


1.0 


(-0.5, 2.5) 


$ 50,000-$59,999 


-1.2 


(-2.4, 0.0) 


$ 60,000-$74,999 


0.7 


(-0.5, 1.8) 


$ 75,000-$99,999 


0.0 


(-0.8, 0.8) 


$100,000 or more 


-0.3 


(-0.9, 0.3) 


Don’t know/refused 


mm 





A single asterisk (”) indicates a NDR significantly different from zero at the 90% confidence 
level. 



A double asterisk (**) indicates a suppressed NDR, or too few cases to accurately compute. 
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TFS-3 for Current Teachers 
Reasons for Differences by Question 



Question/Reason for Difference | 


Count 


Percent 


Cover. Still Teaching? 

Total 

Misunderstood question 
Gave ’usual’ case answer 
Manual/general error 
Category problems 
Other 


£ 

2 

1 

1 

1 

1 


ma 

33.3 

16.7 

16.7 

16.7 

16.7 


1. MAIN activity and OTHER main activity? 






Main activity 






Total 


41 


100.0 


Don’t know 


13 


31.7 


Misunderstood question 


12 


29.3 


Time divided between categories 


4 


9.8 


Forgot/remembered info 


3 


7.3 


Situation changed since responding 


2 


4.9 

A A 


Manual/general error 


2 


4.9 

A A 


Category problems 


2 


4.9 

A 


Other 


1 


2.4 


Misunderstood reference period 


1 


2.4 


Misread question 


1 


2.4 


2. Position as a school EMPLOYEE? 






Total 


£ 


mg 


Time changed since responding 


3 


37.5 


Don’t know 


2 


12.5 


Itinerant teacher - wasn’t sure whether to 






include work at all schools 


1 


25.0 


Misunderstood question 


1 


12.5 


Misunderstood reference period 


1 


12.5 
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Question/Reason for Difference j 


Count 


Percent 


3a. Time as a TEACHER? 






Total 

Amount of time teaching changed since 


IQ 


100.0 


responding 


4 


40.0 


Other 


2 


20.0 


Answered as "total employee" 


1 


10.0 


Amount of time teaching varies 
Itinerant teacher - wasn’t sure which 


1 


10.0 


school referenced 


1 


10.0 


Misunderstood reference period 


1 


10.0 


3b. Teaching assignment? 






Tata! 


12 


100.0 


Misunderstood question 


3 


23.1 


Category problems 


3 


23.1 


Situation changed since responding 


2 


15.4 


Don’t know 


2 


15.4 


Forgot/ remembered info 


1 


7.7 


Manual/general error 


1 


7.7 


FR error 


1 


7.7 


4. Other school assignment? 






Total 


2Q 


100.0 


Don’t know 


9 


45.0 


Missed skip pattem/question 


3 


15.0 


More than one category fits situation 


2 


10.0 


Wasn’t sure what one/more categories meant 


2 


10.0 


Misunderstood question 


2 


10.0 


Other 


1 


5.0 


Category problems 


1 


5.0 
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Question/Reason for Difference 


Count 


Percent 


5a. MAIN teaching assignment at this 
school? 

Total 

Teach both assignments 
Couldn’t distinguish between answer 
categories 
Don’t know 
Misunderstood question 
Main assignment changed since responding 
Wasn’t sure whether to report 
population/subject matter 
Manual/general error 
FR error 


IS 

3 

3 

3 

3 

1 

1 

1 

1 


100.0 

18.8 

18.8 

18.8 

18.8 

6.3 

6.3 

6.3 

6.3 | 


5b. Teach classes in OTHER fields at this school? Yes/No 
response 

Total 

Misunderstood Question 
Didn’t know what to include/exclude 
Forgot/remembered info 
Don’t know 

Missed skip pattern/ question 
Stopped teaching other fields since 
responding 

Stalled teaching other fields since 
responding 

Couldn’t distinguish between answer 
categories 
Don’t know 
Manual/general error 
FR error 
Misread question 
Category problems 


25 

5 

4 

3 

3 

2 

1 

1 

1 

1 

1 

1 

1 

1 


100.0 

20.0 

16.0 

12.0 

12.0 

8.0 

4.0 

4.0 

4.0 

4.0 

4.0 

4.0 

4.0 

4.0 


5b. Teach classes in OTHER fields at this school? 
Field code 






Total 


IQ 


100.0 


Don’t know 


5 


50.0 


Teach both assignments 


3 


30.0 

« A A 1 


Other assignment changed since responding 


1 


10.0 


Wasn’t sure whether to report 




10.0 1 


population/subject matter 


1 
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Question/Reason for Difference 


Count 


Percent 


6a. Teaching certificate in this State in this Main teaching 
assignment field? 

Total 


n 


100.0 


Missed skip pattem/question 


3 


27.3 


Don’t know 


2 


18.2 


Certificate applies to another state 


2 


18.2 


Received certificate since responding 


1 


9.1 


Unsure what "Teaching Certificate" meant 


1 


9.1 


Forgot/remembered information 


1 


9.1 


Manual/general error 


1 


9.1 


6b. Type of Certification? 






Total 


22 


100.0 


Wasn’t sure what one or more categories 
meant 


13 


40.6 


No choice described certification 


5 


15.6 


Don’t know what certification they have 


4 


12.5 


Misunderstood question 


2 


6.3 


Forgot/remembered information 


2 


6.3 


Other 


2 


6.3 


Received certificate since responding 


1 


3.1 


Missed skip pattem/question 


1 


3.1 


Refused to answer in one interview 


1 


3.1 


Don’t know 


1 


3.1 


6c. Certificate granted within the last 12 months? 






Total 


14 


ms 


Misunderstood question 


4 


28.6 


Couldn’t remember when certificate was 
granted 


3 


21.4 


Missed skip pattem/question 


2 


14.3 


FR error 


2 


14.3 


Other 


2 


14.3 


Don’t know 


1 


7.1 




91 



Appendix D-5 
December 27, 1993 



Question/Reason for Difference [ 


Count 1 


Percent 


7a. Teaching certificate in this state in OTHER teaching 
alignment field? 

Total 

Don’t know 

Misunderstood question 
Missed skip pattem/question 
Other 

Realized certificate is in other field 
Realized certificate applies to another 
another state 

Realized certificate is not in other field 
Forgot/remembered info 
FR error 


26 

17 

5 

3 

3 

2 

2 

2 

1 

1 


100.0 

47.2 

13.9 

8.3 

8.3 

5.6 

5.6 

5.6 

2.8 

2.8 


7b. Tvne of certification? 

Total 

No choice described certification 
Don’t know 

Wasn’t sure what one or more categories 
meant 

Refused to answer in one interview 
FR error 


2 

2 

2 

1 

1 

1 


100.0 

28.6 

28.6 

14.3 

14.3 

14.3 


7c. Certificate granted within the last 12 months? 
Don’t know 


1 


100.0 
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| Question/Reason for Difference 


| Count 


| Percent 


8. Grade levels of students in vour classes at THIS school? 






Total 


4 2 


100.0 


Don’t know 


16 


32.7 


Misunderstood question 
I Teaching different students since 


6 


12.2 


responding 

Unsure whether to report grade levels of students sometimes 


4 


8.2 


taught 


4 


8.2 


Misunderstood what "grade level/class" meant 


4 


8.2 


Forgot/remembered info 


4 


8.2 


FR error 

Teach special students - difficulty reporting equivalent 


3 


6.1 


graded levels 


2 


4.1 


| Other 


2 


4.1 


| Misunderstood reference period 

II Teach special students - unsure whether to report as 


2 


4.1 


II "ungraded" or equivalent graded levels 


1 


2.0 


|| Unsure whether to report level of classes with few students 


1 


2.0 


II 9. Community in which this school is located? 






Total 


42 


100.0 


|| Wasn’t sure of population of community 


14 


32.6 


|| More than one category applies 


11 


25.6 


1 Don’t know 


9 


20 9 


| Misunderstood question 


4 


4>v«7 

9.3 


FR error 


2 


4.7 


| Other 


2 


4.7 


| Missed skip pattem/question 


1 


2.3 


10. leaching in the SAME school? 






Total 

Don't know 

1 Taught in two schools last year, one this 


IQ 

3 


me 

30.0 


this year 


3 


30.0 


Misunderstood question 


2 


20.0 


Other 


1 


10.0 


1 Misunderstood reference period 


1 


10.0 
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I Question/Reason for Difference | 


Count 


Percent | 


11. Teaching in same state? Yes/No response 

Unsure what "Schools and Staffing Survey" 
meant 


1 


100.0 


11. Teaching in same state? State code 


no reas 


ons listed 


15. Moved from last vear’s school to this Vgfl r 'S 5ChwI2 
Total 

Don’t know 
Misunderstood question 
Manual/general error 


4 

2 

1 

1 


100.0 

50.0 

25.0 
25.0 


13. Religious affiliation? 


no reasc 


ins listed 


29b(l). Academic base year salary during the CURRENT 
SCHOOL YEAR for teaching in this school? 

Total 

Wasn’t sure of exact amount 
Salary changed since responding 
Don’t know 

Included other salary earnings 
Misunderstood question 
FR error 

Included another source of income 
Forgot/remembered info 
Misunderstood reference period 
Manual/general error 
Wasn’t sure how to report as an itinerant 
teacher 

Gave after-tax earnings 


109 

71 

9 

9 

4 

3 

3 

2 

2 

2 

2 

1 

1 


100.0 

65.1 

8.3 

8.3 

3.7 

2.8 
2.8 
1.8 
1.8 
1.8 
1.8 

0.9 

0.9 
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Question/Reason for Difference 


j Count 


| Percent | 


29b(2). Additional compensation for 

extracunicular/additional activities from school? 
Yes/No response 






Total 


45 


100.0 


Taken on/given up activities since responding 


14 


31.1 


Remembered/forgot activities 


9 


20.0 


Don't know 


8 


17.8 


Unsure what "additional compensation” meant 


5 


11.1 


Other 


2 


4.4 


Missed skip pattem/question 


2 


4.4 


Changed jobs since responding 


1 


2.2 


Misunderstood question 


1 


2.2 


Manual/general error 


1 


2.2 


FR error 


1 


2.2 


Didn’t know what to include/exclude 


1 


2.2 


29b(2). Additional compensation for 

extracunicular/additional activities from school? 
Monetary response 






Total 


24 


100.0* 


Wasn’t sure of exact amount 


13 


54.2 


Remembered/forgot activities 


6 


25.0 


Reported expected earnings 


2 


8.3 


Don’t know 


1 


4.2 


Unsure what "additional compensation" meant 


1 


4.2 | 


Misunderstood question 


1 


4.2 
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| Question/Reason for Difference 


Count 


Percent *j 


| 

29b(3). Additional compensation outside school system? 
Yes/No response 

Total 


4£ 


100.0 


Don’t know 


13 


28.3 


Unsure what constituted job 


7 


15.2 


Took up job since responding 


4 


8.7 


Misunderstood question 


4 


8.7 


Missed skip pattem/quesdon 


3 


6.5 


Misunderstood reference period 


3 


6.5 


Quit job since responding 


2 


4.3 


Other 


2 


4.3 


Forgot/remembered info 


2 


4.3 


FR error 


2 


4.3 


Didn’t know what to include/exclude 


2 


4.3 


Refused to answer in one interview 


1 


2.2 


Manual/general error 


1 


2.2 


29b(3). Additional compensation outside school system? 






Monetary response 




0 


Total 


14 


100.0 


Wasn't sure of exact amount 


8 


57.1 


Reported expected earnings 


2 


14.3 


Don’t know 


2 


14.3 


Refused to answer in one interview 


1 


7.1 


Misunderstood reference period 


1 


7.1 


29b(3a). Job outside the school system? 






Total 


2 


100.0 


Don’t know 


2 


22.2 


Missed skip pattem/question 


2 


22.2 


Forgot/remembered info 


2 


22.2 


Changed jobs since responding 


1 


11.1 


Has more than one Job 


1 


11.1 


Misunderstood question 


1 


11.1 
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Question/Reason for Difference 


| Count 


Percent 


30. Income-in-kind? 






Total 


22 


100.0 


Wasn’t sure what to report as income-in- 






kind 


7 


30.4 


Unsure whether to report partial income- 






in-kind 


4 


17.4 


Don’t know 


3 


13.0 


Income-in-kind changed since responding 


2 


8.7 


Missed skip pattem/qiiestion 


2 


8.7 


Forgot/remembered info 


2 


8.7 


Other 


1 


4.3 


Manual/general error 


1 


4.3 


Misread question 


1 


4.3 


31. Total combined income of ALL FAMILY MEMBERS 






aee 14 and older in vour household durine 1991? 






Total 


M 


100.0 


Unsure of exact amount 


41 


48.8 


Don’t know 


11 


J3.1 


Unsure what to include/exclude 


8 


9.5 


Misunderstood reference period 


7 


8.3 


Wasn’t sure whether to include adult children 


4 


4.8 


FR error 


4 


4.8 


Misunderstood question 


2 


2.4 


Refused to answer in one interview 


2 


2.4 


Other 

i 


1 


1.2 


Missed skip pattern/question 


1 


1.2 


Forgot/remembered info 


1 


1.2 


Manual/general error 


1 


1.2 


Misread question 


1 


1.2 
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TFS-2 for Former Teachers 
After Reconciliation NDRs by Response Categories 



Question/ 

Response Category 


NDR% 


Confidence 
limits 1 


Cover. Still Teaching 






Yes 


** 




No 


** 




l. main OCCUPATIONAL status? 






Working in elementary/secondary school with 


0.8* 


(0.1, 1.4) 


assignment OTHER THAN teaching 


Working in occupation outside elementary/secondary 
education 


-0.8 


(-1.7, 0.2) 


Attending college/university 


-0.2 


(-0.6, 0.3) 


Homemaking/child rearing 


-0.9 


(-2.0, 0.2) 


Retired 


-0.3 


(-1.2, 0.6) 


Disabled 


** 




Other 


1.5* 


(0.2, 2.8) 


2. MAIN school assignment? 






Administrator 


0.0 


(-3.6, 3.6) 


Nonteaching specialist 


mm 




Resource person for other teachers 


mm 




Support staff 


mm 




Coach 


mm 




Other 


mm 




4. Position as an EMPLOYEE? 






Full time employee 


-1.6 


(-4.1, 0.1) 


> = 3/4 time, < full-time 


mm 




> = 1/2 time, < 3/4 time 


mm 




> = 1/4 time, < 1/2 time 


mm 




<1/4 time employee 


mm 





A single asterisk (*) indicates a NDR significantly different from zero at the 90% confidence 
level. 



A double asterisk (**) indicates a suppressed NDR, or too few cases to accurately compute. 
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Question/ 

1 Response Category 


NDR% 


Confidence 
Limits | 


5. Altogether, how much do you usually earn at this 
iob before deduction? 






Hour 


-1.2 


(-3.6, 0.4) 


Day 


** 


Week 


** 




Biweekly 


mm 




Month 


1.8 


(-1.3, 4.9) 


Year 

Other 


l 

oo 


(-4.9, 1.3) 


|| Do not know/refused 


mm 




9. New Degree? 






1 ^ es 
No 


0.5 


(-0.1, 1.1) 


-0.5 


(-1.1, 0.1) 


II 11. What type of degree is it? 






I Associate degree 


mm 




| Bachelor’s 


mm 




| Master’s 


mm 




Education specialist/professional diploma 






Doctorate 


mm 




Professional 


mm 

mm 




12. Major field of study for NEW DEGREE? 






| General education 


mm 




I Subject area education 


mm 




Special education 


mm 




| other education 


mm 




General 


mm 




|| Foreign languages 


mm 




Natural sciences 


mm 




Social sciences 


mm 




|| Other 


mm 




a single asterisk ( ) indicates a NDR sigmticantly different from zero at the 90% confidence 
level. 



A double asterisk (**) indicates a suppressed NDR, or too few cases to accurately compute. 
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Question/ 

Response Category 


NDR% 


Confidence 
Limits 1 


13. For what purpose did you earn this degree? 






To increase salary 


** 


1 


For professional development in current field 


** 


| 


To teach in different field than one taught last year 


** 




For nonteaching position in elementary or secondary 






education 


mm 




For occupation outside elementary or secondary 






education other than current job 


mm 




Other 


mm 




14. Currently enrolled in a degree program? 






No 


-0.5 


(-1.3, 0.4) 


Yes, as a full-time student 


-0.3 


(-1.1, 0.4) 


Yes, as a part-time student 


0.8* 


(0.1, 1.5) 


15. What type of degree are vou pursuing? 






Associate degree 


** 




Bachelor’s 


** 




Master’s 


0.0 


(-3.0, 3.0) 


Education specialist/professional 






diploma 


mm 




Doctorate 


mm 




Professional 


mm 





A single asterisk (*) indicates a NDR significantly different from zero at the 90% confidence 
level. 



A double asterisk (**) indicates a suppressed NDR, or too few cases to accurately compute. 
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Question/ 
Response Category 


NDR% 


Confidence 

Limits 


16. Maior field of studv for decree vou are oursuine? 






General education 


mm 




Subject area education 


mm 




Special education 


mm 




Other education 


mm 




General fields 


mm 




Foreign languages 


mm 




Natural sciences 


mm 




Social sciences 


mm 




All others 


mm 




17. For what miroose are vou pursuing this decree? 






To increase salary 


mm 




For professional development in current field 


2.2 


(-2.5, 7.0) 


To teach in different field than one taught last year 


mm 




For nonteaching position in elementary or secondary 
education 


mm 




For an occupation outside elementary or secondary 






education other than current job 


mm 




Other 


mm 




21a. "Lifetime" teaching certificate? 






Yes 


0.0 


(-1.1, 1.1) 


No 


0.2 


(-1.0, 1.3) 


Don’t know/refused 


mm 


. 


21b. Chance in teacher certification since last year? 






No change 


-2.3* 


(-3.9, -0.8) 


Yes-Certification has lapsed 


1.0* 


(0.1, 2.3) 


Yes-Certification has been upgraded 


mm 




Yes-Certified in different field 


mm 




Other change 


mm 





A single asterisk (*) indicates a NDR significan3^iffer5Itfton^M^^meW^*conndence 
level. 

A double asterisk (**) indicates a suppressed NDR, or too few cases to accurately compute. 
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Question/ 

Response Category 


NDR% 


Confidence 

Limits 


28. Total combined income? 






Less than $10,000 


mm 




$ 10,000 - $14,999 


mm 


_* 


$ 15,000 - $19,999 


0.8 


(-0.3, 1.9) 


$ 20,000 - $24,999 


-1.2* 


(-2.2, -0.3) 


$ 25,000 - $29,999 


2.0* 


(0.8, 3.3) 


$ 30,000 - $34,999 


-0.9 


(-2.3, 0.4) 


$ 35,000 - $39,999 


0.2 


(-1.2, 1.5) 


$ 40,000 - $49,999 


-0.6 


(-2.1, 0.8) 


$ 50,000 - $59,999 


-0.3 


(-1.5, 0.9) 


$ 60,000 - $74,999 


0.5 


(-1.0, 2.0) 


$ 75,000 - $99,999 


-1.1* 


(-2.1, -0.1) 


$100,000 or more 


0.2 


(-0.6, 1.0) 


Other 


mm 




29. Current marital status? 






Married 


0.0 


(-0.6, 0.6) 


Widowed, divorced, or separated 


-0.2 


(-0.7, 0.3) 


Never married 


-0.2 


(-0.7, 0.2) 


Don’t know/refused 


** 




30. Dependent children for more than half of their 






financial suDDort? 






None 


-0.8 


(-1.7, 0.2) 


1 


-0.3 


(-1.4, 0.8) 


2 


0.5 


(-0.5, 1.4) 


3 


0.5 


(-0.1, 1.0) 


4 


mm 




5 


mm 




Other 


mm 





A single asterisk (*) indicates a NDR significantly different from zero at the 90% confidence 
level. 



A double asterisk (**) indicates a suppressed NDR, or too few cases to accurately compute. 
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Question/ 




Confidence 


Response Category 


NDR% 


Limits 


31. Aee of youngest child? 






None 


0.0 


(-1.5, 1.5) 


1 to 5 years 


0.0 


(-1.2, 1.2) 


6 to 10 years 


1.4 


(-0.1, 2.8) 


11 to 15 years 


-1.0 


(-2.4, 0.3) 


16 to 20 years 


-0.3 


(-1.4, 0.7) 


21 to 25 years 


** 




26 to 30 years 


mm 




31 to 35 years 


mm 




36 to 40 years 


mm 




Other 


mm 




32a. Other dependent persons for more than half of 






their financial support? 






Yes 


0.0 


(-0.7, 0.7) 


No 


-0.3 


(-1.1, 0.5) 


Don't know/refused 


mm 


32b. How many persons? 






None 


mm 




1 person 


mm 




2 persons 


mm 




Other 

A ^ J 


mm 





level. 




A double asterisk (**) indicates a suppressed NDR, or too few cases to accurately compute. 
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TFS-2 for Former Teachers 
Reasons for Differences by Question 



Question/Reason for Difference 


Count 


Percent 


Cover. Still Teaching? 






Total 


11 


ma 


Didn’t understand what ’teaching’ meant 


4 


36.4 


Misunderstood question 


2 


18.2 


Temporarily not teaching 


1 

1 


9.1 


Student teaching 


9.1 


Forgot/remembered info 


1 

1 


9.1 

A I 


FR error 


9.1 


Don’t know 


1 


9.1 


1. MAIN OCCUPATION STATUS? 






Total 


22 


100.0 


Occupational status changed since responding 


11 


33.3 


Category problems 


11 


33.3 


Misunderstood question 


4 


12.1 


Other 


3 


9.1 


FR error 


2 


6.1 


Forgot/remembered info 


1 


3.0 


Manual/general error 


1 


3.0 


2. MAIN school assignment? 






Iotal 


2 


ma 


Became position since responding 


1 


50.0 


Don’t know 


1 


50.0 


3a. For whom do vou work? 






Total 


6 


100.0 


Changed jobs since responding 


2 


33.3 


Manual/general error 


2 


33.3 


Didn’t know what to include/exclude 


1 


16.7 


Don’t know 


1 


16.7 


4. Position as an EMPLOYEE? 






3M 


2 


ma 


Amount of time spent working varies 


2 


40.0 


Changed jobs since responding 


1 


20.0 


Other 


. 1 


20.0 


Missed skip pattem/question 


1 


20.0 
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Question/Reason for Difference 


Count 


Percent 


5. Altogether, how much yo you usually earn at this job 






before deduction? 






Total 


51 


100.0 


Wasn’t sure of exact amount 


18 


34.6 


Salary changed since responding 


8 


15.4 


Wasn’t sure what to report 


8 


15.4 


Reported in different units 


4 


7.7 


Changed jobs since responding 


3 


5.8 


Thought question asked for net earnings 


2 


3.8 


Missed skip pattem/question 


2 


3.8 


Other 


1 


1.9 


Don’t know . 


1 


1.9 


Misunderstood question 


1 


1.9 


Refused to answer in one interview 


1 


1.9 


Forgot/remembered info 


1 


1.9 


Manual/general error 


1 


1.9 


FR error 


1 


1.9 


9. New Degree? 






Total 


2 


100.0 


Other 


3 


42.9 


Don’t know 


2 


28.6 


Earned new degree since responding 


1 


14.3 


Had some kind of training in mind 


1 


14.3 


10. When did vou earn this new degree? 






Couldn’t remember exact month/year 


1 


100.0 


11. What type of degree is it? 


no reasons listed 


12. Major field of study for NEW DEGREE? 


no reasons listed 


13. For what purpose did vou earn this degree? 
Category problems 


1 


100.0 
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Question/Reason for Difference 


Count 


Percent 


14. Currently enrolled in a degree program! 






Total 


12 


100.0 


Ended enrollment since responding 


3 


25.0 


FR error 


3 


25.0 


Misunderstood question 


2 


16.7 


Enrolled since responding 


1 


8.3 


Missed skip pattem/question 


1 


8.3 


Manual/general error 


1 


8.3 


Other 


1 


8.3 


15. What tvpe of decree are vou jjursuinc? 






Total 


1 


100.0 | 


Wasn’t sure where degree fit in categories 


2 


66.7 


FR error 


1 


33.3 


16. Mai or field of study for decree vou are oursuing? 






Total 


1 


100.0 


Didn’t know difference between answer categories 


3 


42.9 


Pursuing dual major degree 


1 


14.3 


Changed major since responding 


1 


14.3 


FR error 


1 


14.4 


Misread question 


1 


14.3 


17. For what puroose are vou Dursuinc this decree? 






Total 


11 


100.0 


Both categories equally describe situation 


5 


45.5 


Category problems 


5 


45.5 


Don’t know 


1 


9.1 
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Question/Reason for Difference 


Count 


Percent 


21a. "Lifetime" teaching certificate? 






Total 


2Q 


100.0 


Don’t know 


6 


30.0 


Unsure whether certification was lifetime 


5 


25.0 


Received certificate since responding 


2 


10.0 


Thought certificate was lifetime 


2 


10.0 


Other 


2 


10.0 


Wasn’t sure which state(s) question referred to 


1 


5.0 


Forgot/remembered info 


1 


5.0 


Misunderstood question 


1 


5.0 


21b. Change in teacher certification since last year? 






Total 


15 


100.0 


Misunderstood question 


5 


33.3 


Change since responding 


4 


26.7 


Category problems 


3 


20.0 


Misunderstood reference period 


1 


6.7 


Didn’t know what to include/exclude 


1 


6.7 


Other 


1 


6.7 


28. Total combined income? 






Total 


136 


100.0 


Wasn't sure of exact amount 


67 


49.3 


Wasn’t sure what to include/exclude 


25 


18.4 


Don’t know 


13 


9.6 


Misunderstood reference period 


11 


8.1 


Refused to answer question in one interview 


5 


3.7 


Manual/general error 


4 


2.9 


Misunderstood question 


3 


2.2 


Wasn’t sure whether to include adult children 


2 


1.5 


Other 


2 


1.5 


Forgot/remembered info 


1 


0.7 


Gave "usual" case answer 


1 


0.7 


FR error 


1 


0.7 


Category problems 


1 


0.7 | 
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TFS-2R 

Reasons for Differences by Question 



j Question/Reason for Difference 


Count \ 


Percent 


! 79 Current marital status? I 

Total 

Changed since responding 

Missed skip pattern /question 1 


1 

1 

1 


100.0 

50.0 

50.0 


30. Dependent children for more than half of their financial 
support? 

Total 

Number changed since responding 
Don’t know 

Wasn’t sure whether support was more than half 
Other 

Unsure what "children" meant I 

Misunderstood question _| 


22 

12 

5 

4 

3 

3 

2 | 


100.0 

41.4 

17.2 
13.8 

10.3 
10.3 

6.9 


31. Age of vouneest child? 

Total 

Child had birthday since responding 
Misunderstood question 

Manual/general error | 

Missed skip pattem/question I 

FR error 
Other 


21 

22 

3 

3 

1 

1 

| 1 


100.0 

71.0 

9.7 

9.7 

3.2 

3.2 

3.2 


32a. Other dependent persons for more than half of their 
financial support? 

Total 

Had different ideas what dependent meant 
Unsure what "other persons" meant 
Situation changed since responding 
Other 

Wasn’t sure whether support was more than half 


2 

4 

2 

1 

1 

1 1 


100.0 

44.4 

22.2 

11.1 

11.1 

11.1 


32h. How manv persons? 
Total 


4 


100.0 
1 m a 


Situation changed since responding 


2 

1 A 


50.0 
| 50.0 


different ideas what dependent meant 


1 2 
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I. 



Executive Summary 



A. Purpose 

The 1991-92 Teacher Follow-up Survey (TFS) Reinterview and Extensive 
Reconciliation represents the Bureau’s first attempt to employ an extensive 
structured reconciliation. We used cognitive methods to develop probes, 
which elicited respondents’ reasons for response differences. 

This project’s goals were to identify problematic questions, to identify the 
source of the problems, and to offer suggestions for improving the TFS 
questionnaires. 

This report focuses on the reinterview and extensive reconciliation 
methodology and offers suggestions to improve it. The first report discussed 
the results of the reinterview and offered suggestions to improve the TFS 
questionnaires (Jenkins and Wetzel, 1994). 

B. Major Finding 

Based on expectations from past reinterviews, the 1991-92 TFS Reinterview 
and Extensive Reconciliation produced too few differences between the 
original and reinterview responses and, hence, too few reasons for differences. 

We believe this disappointing result occurred because the reinterview in the 
1991-92 TFS was not independent from the original interview. The results 
suggest that in some cases, reinterviewers did not ignore the original 
responses, reducing the independence between the original interview and the 
reinterview. 

C. Recommendation 

We strongly suggest employing a methodology other than a dependent-type 
reinterview to elicit reasons for response differences. We provide detailed 
descriptions of the following methodologies in Section V, Proposed 
Methodological Revisions: 

• an independent reinterview followed by a third-visit small-scale 
unstructured reconciliation, or 

• an independent reinterview with a large-scale extensive reconciliation 
conducted at the same time using computer-assisted interviewing (CATI 
or CAPI), or 

• a monitored independent CATI reinterview followed by a third-call 
reconciliation. 
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n. Introduction 

Traditionally, reinterviews have been designed for one or more of the following 
purposes: 

• to detect whether interviewers have deliberately falsified data; 

• to evaluate interviewer performance; 

• to estimate response variance; and/or 

• to estimate response bias (Forsman and Schreiner, 1991). 

Many reinterviews performed by the Census Bureau focus on estimating response 
variance. Although measuring response variance exposes inconsistencies in 
respondents’ answers between interviews, it does little to explain why the 
inconsistencies occur. 

Consequently, the 1991-92 TFS Reinterview and Extensive Reconciliation was 
designed with a new objective in mind. It focused on determining the reasons for 
respondent and instrument errors. 

m. Methodology 

A. Description of the 1991-92 TFS Reinterview and Extensive Reconciliation 

The Census Bureau conducted the 1991-92 TFS a year after collecting 
information from teachers in the 1990-91 Schools and Staffing Survey (SASS) 
for the National Center for Education Statistics (NCES). The TFS’ purpose 
was to provide information about teacher attrition and to project teacher 
demand (Faupel et al., 1992). In general, the Census Bureau conducted the 
TFS Reinterview and Extensive Reconciliation two to three weeks after the 
TFS. 

Both the TFS and the TFS Reinterview and Extensive Reconciliation contained 
two components: one for former teachers and another for current teachers. 

Each component had its own questionnaire (the TFS-2 for former teachers and 
the TFS-3 for current teachers), asking primarily different questions. The 
reinterview reasked a subset of questions from the TFS. The NCES chose the 
questions for reinterview. The Census Bureau offered suggestions, favoring 
factual over opinion-seeking or attitudinal questions. 

The TFS Reinterview and Extensive Reconciliation was a dependent-type 
reinterview. The original TFS responses were transcribed onto the reinterview 
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questionnaires so that the reinterviewers could reconcile any response 
differences. 



B. Development of the Extensive Reconciliation Probes 

The use of an extensive reconciliation distinguishes this reinterview from 
others. It contained a series of probes aimed at identifying the reason for 
response differences and a reconciliation question to determine the correct 
response. 

Closed-ended probes offered respondents specific reasons for differences. 

They were not the same from question to question, but tailored to each 
reinterview question. We used closed-ended probes to capture the data 
efficiently. 

If the respondent did not choose one of the closed-ended probes, they were 
asked the open-ended probe: "Or was there some other reason [for the differ- 
ence]?". The open-ended reasons were professionally reviewed and clerically 
coded prior to data entry. 

C. Gross Difference Rate (GDR) 

One of the three measures we used to analyze the data from the TFS 
Reinterview and Extensive Reconciliation, and referenced in this report, is the 
gross difference rate (GDR). The GDR is the proportion of responses that 
differ between the original interview and the reinterview. It provides a rough 
idea of how consistently respondents answer a question. We calculated the 
GDR before reconciliation for the overall question. 

Note: The first report details the sample selection and procedures for conducting the 
1991-92 TFS Reinterview and Extensive Reconciliation. 

IV. Results and Discussion 

The 1991-92 TFS Reinterview and Extensive Reconciliation methodology had a 
number of shortcomings, which we first list below and then discuss in detail. 

• The dependent-type reinterview produced too few differences. 

• The reconciliation produced too many open-ended reasons for differences. 

• The reconciliation produced too many general reasons for differences. 



ERIC 



112 



4 



A. Dependent-Type Reinterview Produced Too Few Differences 

The deper.dent-type reinterviuw produced too few differences. Hence, the 
reconciliation produced too rcw reasons for differences. To demonstrate this 
finding, Section 1 describes the ideal response variance model. Section 2 
compares the 1992 methodology to the ideal to explain the effects departures 
from the ideal could have on response variance. Section 3 compares the 1992 
methodology to the 1989 TFS Reinterview and shows how the dependent-type 
reinterview reduced the number of differences. Finally, Section 4 offers reasons 
why the reinterviewers may have referred to the original responses. 

1. The Ideal Response Variance Model 

The ideal model of a response variance reinterview assumes a model of 
independent replication. With this model, responses have the same expected 
values across trials. The expectation is that identical questions asked under 
the same conditions will suffer f rom the same underlying response error. 
This error is random and may or may not occur in any given interview, 
unless a real change has occurred in the characteristic being measured. 

Given independent replication, differences in respondents’ answers suggest 
that at least one failure occurred at some point in the question-response 
process in at least one of the interviews. Either the questions were 
misunderstood; required information the respondents did not have, could not 
recall, or could not provide in the detail desired; or asked for information 
respondents were not willing to accurately provide. However, even with 
independent replication, a reinterview will not be able to detect consistent 
failures in comprehension, memory retrieval, or willingness to answer the 
questions. 

Given independent replication, questions with high GDRs exhibit problems 
to a greater degree than questions with low GDRs. Questions exhibiting 
high GDRs are of the greatest concern to us because they contribute the 
most to measurement error. 

2. Comparing the 1992 Model to the Ideal 

The design of the 1991-92 TFS Reinterview and Extensive Reconciliation 
departed from the ideal in two major ways: it was neither independent nor 
a perfect replication of the original interview. 
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a. Lack of Independence 

1) Original Response Effect 

The original TFS rc sponses were transcribed onto the 
reinterview questionnaires, making this a dependent-type 
reinterview. Evidence exists from past research that having the 
original responses visible or available to the reinterviewers 
results in fewer differences (Schreiner, 1980; Koons, 1973). 

We feel this lack of independence affected our results more than 
the other effects we consider below. 

2) Memory Effect 

Respondents’ memories may also interfere with independence. 
Respondents may wish to appear consistent, rather than admit 
they misunderstand something. To the extent this occurs, 
reinterview measures will be artificially depressed. Blair and 
Sudman (1993) found between one-third to one-half of 
reinterview respondents reported trying to remember their 
original answers. Whether or not this led to artificially 
suppressed reinterview measures was not studied. 

b. I .ack of Replication 

1) Mode of Administration Differences 

The 1991-92 TFS Reinterview and Extensive Reconciliation was 
administered exclusively by phone, whereas 56 percent of the 
original cases were self-administered. At this time, the 
literature (e.g Tamai and Dillman, 1989; Dillman and Tamai, 
1990; Parmer et al., 1992) seems to suggest that questions 
concerning "subjective" phenomena are more likely to be 
affected by mode differences than questions concerning 
"objective" phenomena. Since the reinterview questions are 
mainly "objective," the mixed mode of administration should 
have less of an effect on the results than might otherwise be the 
case. 

Ever. ,.lth the same mode of administration between the original 
interview and the reinterview, data from the 1991 SASS suggest 
that mail may be more consistent than phone (Bushery, et al., 
1992). However, the SASS did not use a controlled experiment. 
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2) Interviewer Differences 

Field representatives (FRs) conducted the portion, of the original 
interviews that were done by phone. Whereas, supervisory field 
representatives (SFf.s) conducted all of the reinterviews. In this 
case, response variance may reflect interviewer differences, 
rather than purely reflecting respondent or instrument error. 

Even with interviewer differences, a theory is that response 
variance will be lowered with better trained interviewers (i.e., 
SFRs). Therefore, FRs in the original interview followed by 
SFRs in the reinter riew should be more consistent than FRs in 
both the original and reinterview. However, in reality, we feel 
that the effect of interviewer differences, if any, is minimal. 

3. Comparing the 1992 Model to the 1989 Methodology 

Table 1 lists the fourteen questions from the 1991-92 TFS Reinterview and 
Extensive Reconciliation that are the same as those from the 1989 TFS 
Reinterview. All but two of the 1992 questions have before-reconciliation 
GDRs significantly lower than tneir 1989 counterpart at the 90% confidence 
level. An asterisk (*) in the 1992 rate column indicates significance. 

Both surveys used a mixed mode of administration and both were subject to 
the effect respondents’ memories played in the process. Therefore, it is 
unlikely that either of these led to the lower GDRs in 1992. 

However, the 1989 and 1992 surveys had two major differences that could 
have contributed to the lower GDRs in 1992: 

• The 1989 methodology used an independent reinterview, whereas the 
1992 methodology used a dependent-type reinterview combined with a 
reconciliation. 

• The 1989 methodology used FRs in both the original interview and 
reinterview. In contrast, the 1992 procedures used SFRs to conduct the 
reinterview and extensive reconciliation, since FRs are not trained to 
conduct a reconciliation. We hoped that the SFRs would ignore the 
original responses. However, the data suggest that this was not the 
case. 
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Table 1. Before Reconciliation GDRs - 1992 versus 1989 



Question 


1992 

GDR% 


1989 

GDR 

% 


’92# 


’89 

# 


Title 


6a 

6a 


5a 

5a 


TFS-3R: 

Main teaching assignment - Field 

check box 


1.6* 

3.0 


11.2 

3.8 


6b 

6b 


5b 

5b 


Teach classes in other fields - Y/N 

Field 


3.7* 

3.4* 


13.5 

17.3 


7a 


6a 1 


Certificate in state in main assignment field 


1.5* 


7.8 


7b 


6b 1 


Type of certificate 


6.3* 


17.6 


8a 


6a2 


Certificate in state in other assignment field 


13.4* 


24.5 


8b 


6b2 


Type of certificate 


9.0 


10.9 


11 


9 


Teaching is same school 


1.6* 


5.1 


15a 


32b 


Academic base year teaching salary-$ 


14.8* 


28.0 


15b 


32c 


Added compensation from school-$ 


20.4* 


59.0 


2 


1 


TFS-2R: 

Main occupational status 


6.9* 


17.6 


16a 


21a 


Lifetime teaching certificate 


3.3* 


7.7 


1 


7 


TFS-3R&2R: 
Still teaching 1 


1.3* 


6.8 



1 We combined the response counts from the 1992 TFS-3(R) and 2(R) questionnaires to 
correspond to the combined 1989 results for this question. 
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4. Possible Reasons Reinterviewers Refer to the Original Responses 

a. Temptation 

Regardless of their interviewing experience and training, perhaps 
reinterviewers are simply "too tempted" by the original response to do 
anything but use it. 

b. Inadequate Training 

1) Missing Instructions 

Perhaps we don’t properly train reinterviewers to ignore the 
original response. A close inspection of our training manual 
reveals that we were less clear on this issue than intended. 
Nowhere does it clearly instruct the SFRs to ignore the original 
response during the reinterview (Harris, 1992). 

Also, the training manual did not contain instructions to deal 
with the potentially biasing situation of respondents having 
difficulty choosing between two or more categories in the 
original interview. For example, suppose they had difficulty 
deciding whether their community was "A large city" or "A 
suburb of a very large city," but in the end they marked "A 
suburb of a very large city." It is very likely the respondent 
voiced this problem in the reinterview, telling the SFR that he 
or she was having difficulty choosing between these two 
categories. Since one of the categories agrees with the original 
response, it makes sense that the SFRs would choose that one, 
inadvertently biasing the results in favor of the original 
response. 

2) Ignored or Misunderstood Instructions 

A reason for believing that our instructions were either ignored 
or misunderstood is provided by a perplexing piece of data. We 
received approximately 17 percent more reasons for differences 
in respondents’ answers than there were actual differences. In 
these cases, reinterviewers asked respondents the reconciliation 
probes when they weren’t supposed to, although, admittedly, it 
is difficult to understand how or why they did this, since there 
weren’t any differences to reconcile. Also, the reverse situation 
occurred. We have differences with no reason given in 
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approximately 13 percent of the cases. Table 2 lists the counts 
of reasons incorrectly provided or omitted. 



Table 2. Counts of Reasons Incorrectly Provided or Omitted 



Form 


Expected 

Reasons 2 


Reasons 

Incorrectly 

Provided 


Reasons 

Incorrectly 

Omitted 


Total 


1069 


180 


141 


(%) 


100.0 


16.8 


13.2 


TFS-2 


408 


33 


44 


(%) 


100.0 


8.1 


10.8 


TFS-3 


661 


147 


97 


(%) 


100.0 


22.2 


14.7 



B. The Reconciliation Produced Tjo Many Open-ended Responses 

Approximately 54% of the total number of reasons respondents gave for 
differences (1108) were open-ended. This unexpectedly high percentage 
suggests that the series of closed-ended probes did a relatively poor job of 
providing respondents with adequate reasons for differences in their 
responses. 

C. The Reconciliation Produced Too Many General Responses 

An even larger deficiency with the reconciliation probes was that 
respondents did not adequately verbalize the reasons for differences in their 
answers when the closed-ended questions did not apply. Approximately 
43% of the open-ended responses were "don’t know" or "misunderstood 
question." This is a much more serious error than obtaining open-ended 
responses that could be coded to specific reasons. The general responses 
led to the omission of useful data. Table 3 at the end of this section lists the 
counts of the types of open-ended reasons given for differences. 



The count of Expected Reasons equals: 

Total Reasons Given - Reasons Incorrectly Provided + Reasons Incorrectly 
Omitted. 
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Possible reasons for the general responses are: 

• A structured reconciliation may not provide enough encouragement for 
"less verbal" respondents to express themselves. In this setting, 
respondents are only asked to talk at any length if they haven’t chosen 
one of the closed-ended reasons. Some respondents may feel 
uncomfortable in this situation. This may be especially true if the pace 
of the interview is such that respondents are given the impression that 
we don’t really want them to take the time to think about, and 
subsequently, verbalize their answers. 

• Perhaps respondents didn’t adeq-«:ely understand why they were being 
asked to explain the differences in their responses. Therefore, they 
didn’t make the necessary effort to do so. They just opted for "don’t 
know" or "misunderstood question," since these responses probably 
satisfied the reinterviewer while not requiring much effort on their part. 
The study by Blair and Sudman (1993), in which respondent 
perceptions of a reinterview were measured, lends credence to this 
hypothesis. This study demonstrates significant amounts of respondent 
uncertainty about the necessity of reinterviews. 

• Related to the above reasons is inadequate reinterviewer training on our 
part. We should have trained reinterviewers to probe respondents 
further when respondents gave a "don’t know" or "misunderstood 
question" response. On the other hand, perhaps the field staff may 
need more training in probing than we can provide. 

• Perhaps respondents truly "don’t know" the reason for the difference in 
their responses. This is likely to occur when the problem is related to 
the layout of the questionnaire. Jenkins and Dillman (1993) suggest 
that layout problems give rise to measurement errors of which 
respondents aren’t aware. This will also be the case when respondents 
can’t remember what they were thinking as they answered the question. 
Ericsson and Simon (1980) assert that what is remembered, and how 
well, will generally depend critically on the interval between the 
moment of acquisition and the moment of recall. 





11 



Table 3. Counts of Types of Open-ended Reasons for Differences 



Form 


Total 


Type 


"Other" 


"Don’t Know" 


"Misunderstood" 


Coded 3 


Total 


599 


42 


171 


87 


299 


(%) 


100.0 


7.0 


28.5 


14.5 


49.9 


TFS-2 


' 201 


19 


31 


23 


128 


(%) 


100.0 


9.5 


15.4 


11.4 


63.7 


TFS-3 


398 


23 


140 


64 


171 


(%) 


100.0 


5.8 


35.2 


16.1 


43.0 



V. Proposed Methodological Revisions 

In this study, we expanded the traditional reinterview’s purpose by including an 
extended reconciliation that was aimed at finding the underlying causes of a question’s 
problem. Based on our results, we think that this is still a reasonable goal to pursue, 
although we didn’t perfectly achieve it. The first report suggests that despite the 
shortcomings of this study, we were still able to extract meaningful results from the 
data and recommend improvements to some questions. 

We do not endorse any one method to evaluate questionnaires. In our view, most 
methods bring to the analysis different strengths and weaknesses. We believe that in 
the short run certain circumstances may dictate that one method is better than another, 
but in the long run using a combination of techniques (cognitive interviewing, focus 
groups, standard field tests, and so on) is advisable. 

In the following discussion, we present several reinterview/reconciliation methods. 

The methodologies need to be implemented and their results evaluated before they can 
be assessed as truly valid questionnaire design tools. 

A. Independent Reinterview Followed Bv A Third-Visit Small-Scale U nstructured 
Reconciliation 



One possibility is to use an independent, large-scale, probability sample 
reinterview via the same mode as the original interview. The purpose of the 
reinterview will be to identify problem questions. The advantage of relying on 



These are open-ended reasons that were coded to specific reasons and captured as 
such. 
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an independent reinterview is that it will eliminate interviewer bias in the 
response variance data. The advantage of a large-scale probability sample is that 
the results will be generalizc.ole. We recommend using the same mode to 
conduct the reinterview as the original interview to alleviate mode differences as 
a plausible explanation for response variance. 

Once the reinterview is complete and problem questions have been identified, we 
recommend that a small number of problem cases be randomly selected for a 
personal visit unstructured reconciliation. In this visit, respondents will be asked 
to explain the reasons for their response differences using the same kind of 
nonleading probes that are used in cognitive interviews. 

In a standard cognitive interview, respondents often become quite adept at 
verbalizing questionnaire problems because of the special techniques employed. 

At the beginning of the interview, the cognitive interviewer explains that the 
purpose of the research is to learn what problems respondents are having with 
the questionnaire. The cognitive interviewer then trains the respondents to 
verbalize their thoughts during a practice session. Finally, the interviewer 
continuously asks respondents throughout the interview to verbalize their 
thoughts. 

In the unstructured reconciliation, respondents will not be asked to report their 
thoughts while they are in the process of answering the questionnaire. Instead, 
the interviewer will point out that a discrepancy exists in a respondent’s answer 
to the original and reinterview questions. The interviewer will then ask why the 
respondent thinks the discrepancy exists using a probe such as "Can you tell me 
more about that?". The advantage of using this method is that it may encourage 
"less verbal" respondents to express themselves. 

The disadvantages with this method are: 

• As with any small-scale study, the results are not generalizable. • 

• The unstructured reconciliation would take place after the original interview 
and reinterview. As a result, this interview may be subject to memory 
decay on the part of respondents. The third visit will also add to respondent 
burden. 

B. Independent Computer-assisted Reinterview and Extensive Reconciliation 

Another possibility is to conduct the reinterview and extensive reconciliation 
using computed-assisted interviewing (CATI or CAPI). The advantage of using 
CATI/CAPI over a paper questionnaire is that respondents’ original responses 
can be suppressed during the reinterview, allowing total independence. 
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Another advantage is that one need not worry about the physical layout of the 
reinterview and extensive reconciliation probes. In the 1991-92 TFS Reinterview 
and Extensive Reconciliation, these probes needed to conform to the structure of 
a paper booklet. To meet this constraint, we developed a very condensed list of 
probes that weren’t as natural sound.ng. As a result, they were less effective 
than we would have liked. 

If this method is used, we suggest adequately pretesting the reconciliation probes. 
After all, one of the disadvantages of using standardized probes is that they are 
subject to the same problems as any other standardized questions. Consequently, 
we suggest giving more time for probe development. 

The schedule for developing the 1991-92 TFS Reinterview and Extensive 
Reconciliation questionnaires was hurried. It allowed very little time for 
developing the probes, let alone pretesting them. In effect, the open-ended 
responses to the survey questionnaire can serve as a pretest for developing future 
reconciliation probes. They can be used to develop a more comprehensive list of 
closed-ended responses. 

We also suggest taking a number of steps to help respondents express themselves 
more specifically. 

• Provide the right environment for respondents, one in which they fully 
understand why we are conducting the reinterview and feel comfortable 
explaining their questionnaire problems. We explain the reason for the 
survey in the introduction, but it is likely that respondents don’t really 
comprehend it. Research demonstrates that respondents often don’t 
understand the purpose of a survey, but will respond to it nonetheless 
(Fowler and Mangione, 1990). It is also possible that they forget the reason 
by the time we get to the reconciliation portion. Perhaps we should remind 
them again just before we begin the reconciliation phase. 

• Remove the "don’t know" answer category from the questionnaire and train 
reinterviewers to probe respondents when they give some other general 
reason, such as "misunderstood question." In this case, reinterviewers can 
ask a probe, such as "Can you tell me more specifically what it is about the 
question that you misunderstood?". 

• Train the reinterviewers to sufficiently slow down when they ask the open- 
ended reconciliation probes to give respondents the correct impression that 
we want them to take the time to think about and verbalize their answers. 

There are still several disadvantages with this methodology: 



ERJC 



122 



14 



• In those cases in which the original interview was conducted by mail, mode 
differences will exist. In these cases, one could never fully rule out mode 
as a reason for differences. 

• This method does not provide much opportunity for "non-verbal" 
respondents to gain proficiency at expressing themselves. However, if the 
closed-ended probes are properly designed, this should matter less. 

• This method is unlikely, under any circumstances, to capture problems that 
are related to the layout of the questionnaire. Respondents can not explain a 
problem of which they are unaware. 

• The recommendations we’ve made-pretesting the reconciliation probes and 
using C ATI— require an investment that may not be cost effective initially. 
However, if attention is given tc developing this method, there may come a 
time when it provides useful, generalizable data of the sort not obtainable 
from other questionnaire evaluation methods. 

C. Monitored Independent CATI Reinterview Followed by a Third-Call 
Reconciliation 

Another method is to combine a CATI reinterview with audio taping and 
behavior coding. An experiment in the 1991 American Housing Survey 
used a less refined version of this technique (Waite, 1993). CATI 
interviewers performed the reinterview of the original cases completed in the 
field. The SFRs monitored the CATI reinterview. The SFRs used forms 
with the transcribed responses from the original interview to note 
discrepancies in respondents’ answers. They also noted the behaviors of 
respondents and the CATI interviewers. After monitoring the reinterview, 
the SFR contacted the respondent again and performed a structured 
reconciliation. 

A variation to this method would be to use cognitive interviewers in place of 
SFRs. These experts could then perform an unstructured reconciliation 
rather than a structured one. 

We recommend audio taping reinterviews in the CATI centers instead of in 
personal visits or from the reinterviewers’ homes (CAPI). Presently, the 
Survey of Income and Program Participation is taping reinterviews in 
respondents’ homes with only a 61 percent success rate. 
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This method has two of the same disadvantages as given for the previous 
method: 

• When the original interview is conducted by mail, mode differences 
may exist. 

• This method is unlikely to capture questionnaire layout problems. 
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